Repository: mne-tools/mne-python

Issue:
DOC: Actively suggest updating MNE to a supported version
I often have people come to me locally with bugs, only to find that they are on some old version of MNE-Python. I think it's not generally known to neuroscience people that upgrading is important for bug fixes etc., so it would be good to make it clear what the software support policy is and that upgrading is important.

We should document this clearly as much as possible, but I have two other ideas that will actively "pop up" / ping users more actively:

1. Add a date-based warning *on import*. For example, if we release 1.6 on Nov 1 2023, add to `mne/__init__.py` code to emit a warning on Nov 1 2024 that a new version of MNE is available and that they should update to avoid bugs (or some similar explicit message). We don't absolutely know or guarantee our release cadence, but the longest we have gone between releases in the last few years is probably like ~8 mo, and I think we now shoot for ~4 mo -- so to be safe if can make it year from when we cut a release. This could be controllable with a `MNE_CHECK_VERSION_DATE` env var or something in case people have wacky datetime settings.
2. Query the current release version over the internet when `mne.sys_info` is called with a new kwarg `check_version=True`. We don't want our software to start trying to "phone home" without a user asking in some way (i.e., definitely *not* at import time). Then the question becomes what should the default value of `check_version` in `mne.sys_info(..., check_version=...)` be:

   - `False`: least invasive but also least informative for users. In our bug report forms and forum stuff we ask people to call `mne sys_info --check-version` or `mne.sys_info(check_version=True)` or so. But the probability of users seeing an informative warning goes down.
   - `True`: most invasive but also most informative for users.
   - `None`: in between if we make it mean "try to check and if it takes more than 2 sec don't bother" or something. (FWIW, `mne sys_info` on my machine currently takes ~3.7s.)

   Whichever way we go, the default behavior for `None` could be controllable with `MNE_SYS_INFO_CHECK_VERSION` var in case people want the default `None` behavior to mean `False` or `True`.

Happy to hear thoughts on these and other ideas for suggesting people update to supported versions.


Discussion:
+1, I often have people come to me with bugs already fixed.
I'd guess that the calendar-based approach would probably do a better job of *preventing* the situation you describe @larsoner, as it seems more likely to get people to *proactively* stay up to date. But it's also more invasive so I'm a bit wary of annoying people --- one of my pet peeves with R is the ~7 lines of output it spits on your console every time you fire it up; to do this well we'd probably need at least 3 lines: the warning, guidance on how to update (even if just a URL), and guidance on how to silence the version check.

if we take the sys_info approach, I'm also happy because it's more accurate, and easy to do:

```py
import json
from packaging import version
from urllib.request import urlopen

url = "https://api.github.com/repos/mne-tools/mne-python/releases/latest"
with urlopen(url) as f:
    response = json.load(f)
print(version.parse(response["tag_name"]))
```
> to do this well we'd probably need to include not just the warning but also guidance on how to update (even if just a URL) and guidance on how to silence the version check.

I think both of these are pretty easily doable. For the first we should point to our updating page URL, and for the second we can add a config/env var.
I'm :-1: on the import warning, because this impacts reproducibility if a perfectly fine script suddenly starts to emit warnings in the future. The chance of annoying people is also pretty high IMO, so controlling that with an env var is not a good solution either.

I like the `mne.sys_info()` approach though, but if we go for `check_version=True` by default, we should really introduce something like a deprecation cycle, otherwise people might run into the same reproducibility issues when they use `mne.sys_info()` in their scripts.
I agree with @cbrnr that this impacts reproducibility. Especially for users who do not want to update MNE within a project (e.g. because they cannot easily re-run all analyses with the new version due to their computational resources).

It seems to me as this is mostly useful if a user runs into a bug - so maybe it's really about education i.e. documentation? It seems like an easy first step could be to update our BUG template on GitHub to prominently tell the user to check behavior with the latest version.
> I think both of these are pretty easily doable.

sure, easily doable. I was just making the point that for folks (like me) who are annoyed by a bunch of extra terminal output, it will be for us at least 3 lines.

> if we go for check_version=True by default, we should really introduce something like a deprecation cycle, otherwise people might run into the same reproducibility issues

I'm not sure this point about the deprecation cycle makes sense; if we start with `check_version=None` then *all* users will see a warning when they run `sys_info` unless/until they update their script to say `check_version=False`. On the other hand, if we start out with `check_version=True` then *only* users on old versions will see a warning, and even then *only* once 1.6 (current dev) becomes "old".
> It seems to me as this is mostly useful if a user runs into a bug

That's part of it. But also our code will always have some fraction of hidden silent bugs, so people hit these unknowingly. So the warning will also help people who hit those and don't know it. Someone can think their script is perfectly fine, and while *their code* could be, *our software* could have some bug that they never know is fixed, so they don't bother to update and live with some errant behavior unknowingly.

But it sounds like others think the cost (e.g., having to set an env var to avoid a warning) isn't worth the potential benefits here so I'll drop (1).

Sounds like people are generally in favor of some form of (2) so I'll probably give that one a shot for 1.6.
API idea:
```py
mne.sys_info(...,
             check_version = False,   # never check
             check_version = "info",  # print as part of the sys_info output
             check_version = "warn",  # same as "info" but also log a warning if outdated
             )
```
maybe with `True` being an alias for "info"?  Add'l output might be something like
```
Core
├☑ mne               1.6.0 (outdated, current stable is 1.8.1)
```
> > I think both of these are pretty easily doable.
> 
> 
> 
> sure, easily doable. I was just making the point that for folks (like me) who are annoyed by a bunch of extra terminal output, it will be for us at least 3 lines.
> 
> 
> 
> > if we go for check_version=True by default, we should really introduce something like a deprecation cycle, otherwise people might run into the same reproducibility issues
> 
> 
> 
> I'm not sure this point about the deprecation cycle makes sense; if we start with `check_version=None` then *all* users will see a warning when they run `sys_info` unless/until they update their script to say `check_version=False`. On the other hand, if we start out with `check_version=True` then *only* users on old versions will see a warning, and even then *only* once 1.6 (current dev) becomes "old".

Yes but let's be honest, we're constantly breaking user code and the short deprecation cycles we currently have are more an alibi to soothe ourselves (the devs) than really helping users actively using our software. The cycles are waaaay too short, for example in a "production" environment it's not unusual to skip one or maybe even two or three versions before updating again, and then your scripts are already non-functional because you missed the deprecation warnings.
We could fully embrace SemVer and bump the major version every time the API has breaking changes. 
> Yes but let's be honest, we're constantly breaking user code and the short deprecation cycles we currently have are more an alibi to soothe ourselves (the devs) than really helping users actively using our software.

Just wanted to chime in that as a _user_ I've experienced this. For example in [plot_topomap](https://mne.tools/stable/generated/mne.viz.plot_topomap.html) the `show_names` parameter was deprecated in `1.2` ~~and switched to~~ in favour of `names`, and dropped in `1.3`. IIRC the deprecation cycle was like 2 months so I think I just missed it entirely and only noticed when my code for a project broke after updating. It was a super easy fix (for me) but I remember having to check the doc across versions to figure out what happened.

But maybe that's a digression from this ticket. Anyways I think that as a _user_, some mechanism in `sys_info` that communicates that my version is outdated would probably come in handy.




I like the idea to include a message in the output of `mne.sys_info()`, no warnings or depreciations necessary.
I also like the idea of adding that information to sys_info(). But simply as an info and not immediately as a nudge to upgrade 😃

Resolving patch:
diff --git a/mne/commands/mne_sys_info.py b/mne/commands/mne_sys_info.py
--- a/mne/commands/mne_sys_info.py
+++ b/mne/commands/mne_sys_info.py
@@ -41,6 +41,13 @@ def run():
         action="store_false",
         default=True,
     )
+    parser.add_option(
+        "--no-check-version",
+        dest="check_version",
+        help="Disable MNE-Python remote version checking.",
+        action="store_false",
+        default=True,
+    )
     options, args = parser.parse_args()
     dependencies = "developer" if options.developer else "user"
     if len(args) != 0:
@@ -51,6 +58,7 @@ def run():
         show_paths=options.show_paths,
         dependencies=dependencies,
         unicode=options.unicode,
+        check_version=options.check_version,
     )
 
 
diff --git a/mne/utils/config.py b/mne/utils/config.py
--- a/mne/utils/config.py
+++ b/mne/utils/config.py
@@ -14,9 +14,13 @@
 import subprocess
 import sys
 import tempfile
-from functools import partial
+from functools import lru_cache, partial
 from importlib import import_module
 from pathlib import Path
+from urllib.error import URLError
+from urllib.request import urlopen
+
+from packaging.version import parse
 
 from ._logging import logger, warn
 from .check import _check_fname, _check_option, _check_qt_version, _validate_type
@@ -569,6 +573,7 @@ def _get_numpy_libs():
 print(gi.renderer)"""
 
 
+@lru_cache(maxsize=1)
 def _get_gpu_info():
     # Once https://github.com/pyvista/pyvista/pull/2250 is merged and PyVista
     # does a release, we can triage based on version > 0.33.2
@@ -581,7 +586,14 @@ def _get_gpu_info():
     return out
 
 
-def sys_info(fid=None, show_paths=False, *, dependencies="user", unicode=True):
+def sys_info(
+    fid=None,
+    show_paths=False,
+    *,
+    dependencies="user",
+    unicode=True,
+    check_version=True,
+):
     """Print system information.
 
     This function prints system information useful when triaging bugs.
@@ -600,9 +612,16 @@ def sys_info(fid=None, show_paths=False, *, dependencies="user", unicode=True):
         Include Unicode symbols in output.
 
         .. versionadded:: 0.24
+    check_version : bool | float
+        If True (default), attempt to check that the version of MNE-Python is up to date
+        with the latest release on GitHub. Can be a float to give a different timeout
+        (in sec) from the default (2 sec).
+
+        .. versionadded:: 1.6
     """
     _validate_type(dependencies, str)
     _check_option("dependencies", dependencies, ("user", "developer"))
+    _validate_type(check_version, (bool, "numeric"), "check_version")
     ljust = 24 if dependencies == "developer" else 21
     platform_str = platform.platform()
     if platform.system() == "Darwin" and sys.version_info[:2] < (3, 8):
@@ -695,6 +714,7 @@ def sys_info(fid=None, show_paths=False, *, dependencies="user", unicode=True):
         unicode = unicode and (sys.stdout.encoding.lower().startswith("utf"))
     except Exception:  # in case someone overrides sys.stdout in an unsafe way
         unicode = False
+    mne_version_good = True
     for mi, mod_name in enumerate(use_mod_names):
         # upcoming break
         if mod_name == "":  # break
@@ -719,7 +739,16 @@ def sys_info(fid=None, show_paths=False, *, dependencies="user", unicode=True):
         except Exception:
             unavailable.append(mod_name)
         else:
-            out(f"{pre}☑ " if unicode else " + ")
+            mark = "☑" if unicode else "+"
+            mne_extra = ""
+            if mod_name == "mne" and check_version:
+                timeout = 2.0 if check_version is True else float(check_version)
+                mne_version_good, mne_extra = _check_mne_version(timeout)
+                if mne_version_good is None:
+                    mne_version_good = True
+                elif not mne_version_good:
+                    mark = "☒" if unicode else "X"
+            out(f"{pre}{mark} " if unicode else f" {mark} ")
             out(f"{mod_name}".ljust(ljust))
             if mod_name == "vtk":
                 vtk_version = mod.vtkVersion()
@@ -747,6 +776,9 @@ def sys_info(fid=None, show_paths=False, *, dependencies="user", unicode=True):
                     out(" (OpenGL unavailable)")
                 else:
                     out(f" (OpenGL {version} via {renderer})")
+            elif mod_name == "mne":
+                out(f" ({mne_extra})")
+            # Now comes stuff after the version
             if show_paths:
                 if last:
                     pre = "   "
@@ -756,3 +788,42 @@ def sys_info(fid=None, show_paths=False, *, dependencies="user", unicode=True):
                     pre = " | "
                 out(f'\n{pre}{" " * ljust}{op.dirname(mod.__file__)}')
             out("\n")
+
+    if not mne_version_good:
+        out(
+            "\nTo update to the latest supported release version to get bugfixes and "
+            "improvements, visit "
+            "https://mne.tools/stable/install/updating.html\n"
+        )
+
+
+def _get_latest_version(timeout):
+    # Bandit complains about urlopen, but we know the URL here
+    url = "https://api.github.com/repos/mne-tools/mne-python/releases/latest"
+    try:
+        with urlopen(url, timeout=timeout) as f:  # nosec
+            response = json.load(f)
+    except URLError as err:
+        # Triage error type
+        if "SSL" in str(err):
+            return "SSL error"
+        elif "timed out" in str(err):
+            return f"timeout after {timeout} sec"
+        else:
+            return f"unknown error: {str(err)}"
+    else:
+        return response["tag_name"].lstrip("v") or "version unknown"
+
+
+def _check_mne_version(timeout):
+    rel_ver = _get_latest_version(timeout)
+    if not rel_ver[0].isnumeric():
+        return None, (f"unable to check for latest version on GitHub, {rel_ver}")
+    rel_ver = parse(rel_ver)
+    this_ver = parse(import_module("mne").__version__)
+    if this_ver > rel_ver:
+        return True, f"devel, latest release is {rel_ver}"
+    if this_ver == rel_ver:
+        return True, "latest release"
+    else:
+        return False, f"outdated, release {rel_ver} is available!"


Test patch:
diff --git a/mne/commands/tests/test_commands.py b/mne/commands/tests/test_commands.py
--- a/mne/commands/tests/test_commands.py
+++ b/mne/commands/tests/test_commands.py
@@ -538,7 +538,7 @@ def test_sys_info():
     with ArgvSetter((raw_fname,)):
         with pytest.raises(SystemExit, match="1"):
             mne_sys_info.run()
-    with ArgvSetter() as out:
+    with ArgvSetter(("--no-check-version",)) as out:
         mne_sys_info.run()
     assert "numpy" in out.stdout.getvalue()
 
diff --git a/mne/utils/tests/test_config.py b/mne/utils/tests/test_config.py
--- a/mne/utils/tests/test_config.py
+++ b/mne/utils/tests/test_config.py
@@ -1,15 +1,21 @@
 import os
 import platform
+import re
+from functools import partial
 from pathlib import Path
+from urllib.error import URLError
 
 import pytest
 
+import mne
+import mne.utils.config
 from mne.utils import (
     ClosingStringIO,
     _get_stim_channel,
     get_config,
     get_config_path,
     get_subjects_dir,
+    requires_good_network,
     set_config,
     set_memmap_min_size,
     sys_info,
@@ -95,7 +101,7 @@ def test_config(tmp_path):
 def test_sys_info():
     """Test info-showing utility."""
     out = ClosingStringIO()
-    sys_info(fid=out)
+    sys_info(fid=out, check_version=False)
     out = out.getvalue()
     assert "numpy" in out
 
@@ -109,7 +115,7 @@ def test_sys_info_qt_browser():
     """Test if mne_qt_browser is correctly detected."""
     pytest.importorskip("mne_qt_browser")
     out = ClosingStringIO()
-    sys_info(fid=out)
+    sys_info(fid=out, check_version=False)
     out = out.getvalue()
     assert "mne-qt-browser" in out
 
@@ -134,3 +140,67 @@ def test_get_subjects_dir(tmp_path, monkeypatch):
     monkeypatch.setenv("HOME", str(tmp_path))
     monkeypatch.setenv("USERPROFILE", str(tmp_path))  # Windows
     assert str(get_subjects_dir("~/foo")) == str(subjects_dir)
+
+
+@pytest.mark.slowtest
+@requires_good_network
+def test_sys_info_check_outdated(monkeypatch):
+    """Test sys info checking."""
+    # Old (actually ping GitHub)
+    monkeypatch.setattr(mne, "__version__", "0.1")
+    out = ClosingStringIO()
+    sys_info(fid=out, check_version=10)
+    out = out.getvalue()
+    assert "(outdated, release " in out
+    assert "updating.html" in out
+
+    # Timeout (will call urllib.open)
+    out = ClosingStringIO()
+    sys_info(fid=out, check_version=1e-12)
+    out = out.getvalue()
+    assert re.match(".*unable to check.*timeout.*", out, re.DOTALL) is not None
+    assert "updating.html" not in out
+
+
+def test_sys_info_check_other(monkeypatch):
+    """Test other failure modes of the sys info check."""
+
+    def bad_open(url, timeout, msg):
+        raise URLError(msg)
+
+    # SSL error
+    out = ClosingStringIO()
+    with monkeypatch.context() as m:
+        m.setattr(mne.utils.config, "urlopen", partial(bad_open, msg="SSL: CERT"))
+        sys_info(fid=out)
+    out = out.getvalue()
+    assert re.match(".*unable to check.*SSL.*", out, re.DOTALL) is not None
+
+    # Other error
+    out = ClosingStringIO()
+    with monkeypatch.context() as m:
+        m.setattr(mne.utils.config, "urlopen", partial(bad_open, msg="foo bar"))
+        sys_info(fid=out)
+    out = out.getvalue()
+    match = re.match(".*unable to .*unknown error: .*foo bar.*", out, re.DOTALL)
+    assert match is not None
+
+    # Match
+    monkeypatch.setattr(
+        mne.utils.config,
+        "_get_latest_version",
+        lambda timeout: "1.5.1",
+    )
+    monkeypatch.setattr(mne, "__version__", "1.5.1")
+    out = ClosingStringIO()
+    sys_info(fid=out)
+    out = out.getvalue()
+    assert " 1.5.1 (latest release)" in out
+
+    # Devel
+    monkeypatch.setattr(mne, "__version__", "1.6.dev0")
+    out = ClosingStringIO()
+    sys_info(fid=out)
+    out = out.getvalue()
+    assert "devel, " in out
+    assert "updating.html" not in out