Repository: sunpy/sunpy

Issue:
Rethinking and rewriting sunpy.self_test
We are currently using astropy's test runner for `sunpy.self_test` this was really designed for setup.py and is therefore very full of features which are probably not needed for self_test.

Before we (I) go deleting swathes of code as I love to do. What do we want to achieve with self test? Is a very slim wrapper around `pytest --pyargs sunpy` all we need?


Discussion:
Do we need even need a self test?
That's a valid question. People do use it, and it's a potentially useful debugging tool.
Isn't that only because we tell them to in the guide? 
> That's a valid question. People do use it, and it's a potentially useful debugging tool.

Just for sake of more info,
In issue #3369 , self test were used to know import errors with conda forge. Is there any other ways of testing that sunpy will do as expected in all environments and OS?
I remember someone mentioned that `self_test()` can be used to check that a SunPy installation isn't broken.
The graphs in #3818 made me think about this again. Importing the runner (which happens at `import sunpy` time, so every time) takes a decent chunk of our import time, so moving to something lighter would have a real benefit.

Underneath *layers* of stuff the astropy test runner is just calling [`pytest.main`](https://github.com/astropy/astropy/blob/master/astropy/tests/runner.py#L248). What I therefore propose is we write a function (inside `sunpy.tests` so affiliated packages can also use it) which is a very minimal `self_test`. What I mean by that is I think it should support a `online=False` option, but otherwise no other options, i.e.:

* No support for coverage.
* No figure tests.
* No support for only running subpackages (i.e. it always runs all the tests)

The command should also happily run with only the `pytest-astropy` package installed as an extra dependency for running the tests. [This is more of a generic testing policy that all tests which depend of something not installed by `pytest-astropy` should skip if they can't run?]
> * No support for only running subpackages (i.e. it always runs all the tests)

This would mean making sure that the tests fail nicely if the extra packages are not installed. 
Hi! I am sorry if this sound wierd but I wanted to have a look at the code related to this issue, but I couldn't find it. Could anyone please guide me towards it?
> Hi! I am sorry if this sound wierd but I wanted to have a look at the code related to this issue, but I couldn't find it. Could anyone please guide me towards it?

I think it is this file: https://github.com/sunpy/sunpy/blob/master/sunpy/tests/runner.py

Resolving patch:
diff --git a/sunpy/__init__.py b/sunpy/__init__.py
--- a/sunpy/__init__.py
+++ b/sunpy/__init__.py
@@ -11,7 +11,7 @@
 import sys
 import logging
 
-from sunpy.tests.runner import SunPyTestRunner
+from sunpy.tests.self_test import self_test
 from sunpy.util import system_info
 from sunpy.util.config import load_config, print_config
 from sunpy.util.logger import _init_log
@@ -50,8 +50,6 @@ def _get_bibtex():
 
 __citation__ = __bibtex__ = _get_bibtex()
 
-self_test = SunPyTestRunner.make_test_runner_in(os.path.dirname(__file__))
-
 # Load user configuration
 config = load_config()
 


Test patch:
diff --git a/sunpy/coordinates/tests/test_frameattributes.py b/sunpy/coordinates/tests/test_frameattributes.py
--- a/sunpy/coordinates/tests/test_frameattributes.py
+++ b/sunpy/coordinates/tests/test_frameattributes.py
@@ -23,7 +23,7 @@ def oca():
 
 
 def test_now(attr):
-    """ We can't actually test the value independantly """
+    """ We can't actually test the value independently """
     result, converted = attr.convert_input('now')
 
     assert isinstance(result, Time)
@@ -31,7 +31,7 @@ def test_now(attr):
 
 
 def test_none(attr):
-    """ We can't actually test the value independantly """
+    """ We can't actually test the value independently """
     result, converted = attr.convert_input(None)
 
     assert result is None
diff --git a/sunpy/io/tests/test_fits.py b/sunpy/io/tests/test_fits.py
--- a/sunpy/io/tests/test_fits.py
+++ b/sunpy/io/tests/test_fits.py
@@ -91,10 +91,12 @@ def test_write_with_metadict_header_astropy(tmpdir):
     with pytest.warns(SunpyUserWarning, match='The meta key comment is not valid ascii'):
         sunpy.io.fits.write(str(temp_file), data, meta_header)
     assert temp_file.exists()
-
+    fits_file.close()
 
 # Various warnings are thrown in this test, but we just want to check that the code
 # works without exceptions
+
+
 @pytest.mark.filterwarnings('ignore')
 def test_fitsheader():
     """Test that all test data can be converted back to a FITS header."""
diff --git a/sunpy/tests/helpers.py b/sunpy/tests/helpers.py
--- a/sunpy/tests/helpers.py
+++ b/sunpy/tests/helpers.py
@@ -1,4 +1,3 @@
-import os
 import platform
 import warnings
 from pathlib import Path
@@ -81,7 +80,7 @@ def figure_test(test_function):
     in the library. A PNG is also created in the 'result_image' directory,
     which is created on the current path.
 
-    All such decorated tests are marked with `pytest.mark.figure` for convenient filtering.
+    All such decorated tests are marked with `pytest.mark.mpl_image` for convenient filtering.
 
     Examples
     --------
@@ -102,49 +101,9 @@ def test_wrapper(*args, **kwargs):
         if ret is None:
             ret = plt.gcf()
         return ret
-
     return test_wrapper
 
 
-# Skip coverage on this because we test it every time the CI runs --coverage!
-def _patch_coverage(testdir, sourcedir):  # pragma: no cover
-    """
-    This function is used by the ``setup.py test`` command to change the
-    filepath of the source code from the temporary directory "setup.py"
-    installs the code into to the actual directory "setup.py" was executed in.
-    """
-    import coverage
-
-    coveragerc = os.path.join(os.path.dirname(__file__), "coveragerc")
-
-    # Load the .coverage file output by pytest-cov
-    covfile = os.path.join(testdir, ".coverage")
-    cov = coverage.Coverage(covfile, config_file=coveragerc)
-    cov.load()
-    cov.get_data()
-
-    # Change the filename for the datafile to the new directory
-    if hasattr(cov, "_data_files"):
-        dfs = cov._data_files
-    else:
-        dfs = cov.data_files
-
-    dfs.filename = os.path.join(sourcedir, ".coverage")
-
-    # Replace the testdir with source dir
-    # Lovingly borrowed from astropy (see licences directory)
-    lines = cov.data._lines
-    for key in list(lines.keys()):
-        new_path = os.path.relpath(
-            os.path.realpath(key),
-            os.path.realpath(testdir))
-        new_path = os.path.abspath(
-            os.path.join(sourcedir, new_path))
-        lines[new_path] = lines.pop(key)
-
-    cov.save()
-
-
 def no_vso(f):
     """
     Disable the VSO client from returning results via Fido during this test.
diff --git a/sunpy/tests/runner.py b/sunpy/tests/runner.py
deleted file mode 100644
--- a/sunpy/tests/runner.py
+++ /dev/null
@@ -1,130 +0,0 @@
-import os
-
-from astropy.tests.runner import TestRunner, keyword
-
-
-class SunPyTestRunner(TestRunner):
-    """
-    A runner for SunPy tests, it modifies the arguments to the astropy runner
-    to maintain a similar but more SunPy focused CLI.
-    """
-    # Disable certain astropy flags
-    @keyword()
-    def remote_data(self, remote_data, kwargs):
-        return NotImplemented  # pragma: no cover
-
-    # Change the docsting on package
-    @keyword(priority=10)
-    def package(self, package, kwargs):
-        """
-        package : `str`, optional
-            The name of a specific package to test, e.g. 'map' or 'net.vso'.
-            If nothing is specified all default SunPy tests are run.
-        """
-        return super().package(package, kwargs)
-
-    # Define our own options
-    @keyword(False, priority=3)
-    def online(self, online, kwargs):
-        """
-        online : `bool`, optional
-            Enable the online tests if `True` or disable them if `False`.
-        """
-        if online:
-            return ['--remote-data=any']
-
-        return []
-
-    @keyword(False, priority=2)
-    def online_only(self, online_only, kwargs):
-        """
-        online_only: `bool`, optional
-            If `True` only online tests are run.
-        """
-        r = []
-        if online_only:
-            r.append('-k remote_data')
-            if not kwargs['online']:
-                r.append('--remote-data=any')
-
-        return r
-
-    @keyword(False)
-    def figure(self, figure, kwargs):
-        """
-        figure : `bool`, optional
-            Enable the figure tests.
-        """
-        if not figure and not kwargs['figure_only']:
-            return ['-m', 'not figure']
-
-        return []
-
-    @keyword(False)
-    def figure_only(self, figure, kwargs):
-        """
-        figure_only : `bool`, optional
-            Only run the figure tests.
-        """
-        if figure:
-            return ['-m', 'figure']
-
-        return []
-
-    @keyword()
-    def figure_dir(self, figure_dir, kwargs):
-        """
-        figure_tests : `str`, optional
-            Set the output directory for figure test images and hashes.
-        """
-        # If our test path is outside of our base dir (docs) then we have to
-        # skip sending --figure_dir as we will not have hit the conftest.py
-        # file.
-        if kwargs['test_path'] and self.base_path not in kwargs['test_path']:
-            return []
-
-        if figure_dir:
-            return ['--figure_dir', figure_dir]
-
-        return []
-
-    # Define this to change the default value to None
-    @keyword()
-    def plugins(self, plugins, kwargs):
-        """
-        plugins : `list`, optional
-            Plugins to be passed to ``pytest.main`` in the ``plugins`` keyword
-            argument.
-        """
-        # Plugins are handled independently by `run_tests` so we define this
-        # keyword just for the docstring
-        return []
-
-    @keyword()
-    def coverage(self, coverage, kwargs):
-        if coverage:
-            coveragerc = os.path.join(self.base_path, "tests", "coveragerc")
-            ret = []
-            for path in self.package_path:
-                ret += ["--cov", path, "--cov-config", coveragerc]
-            return ret
-
-        return []
-
-    @keyword()
-    def cov_report(self, cov_report, kwargs):
-        if kwargs['coverage'] and cov_report:
-            a = [cov_report] if isinstance(cov_report, str) else []
-            return ['--cov-report'] + a
-
-        return []
-
-    @keyword()
-    def docs_path(self, docs_path, kwargs):
-        """
-        Ignore the docs directory if we have set test_path.
-        """
-        if kwargs['test_path']:
-            return []
-        else:
-            return super().docs_path(docs_path, kwargs)
diff --git a/sunpy/tests/self_test.py b/sunpy/tests/self_test.py
new file mode 100644
--- /dev/null
+++ b/sunpy/tests/self_test.py
@@ -0,0 +1,116 @@
+import importlib
+
+from pkg_resources import get_distribution
+
+
+def find_missing_dependencies(package="sunpy", extras=None):
+    """
+    List missing dependencies.
+
+    Given a package and, optionally, a tuple of extras, identify any packages
+    which should be installed to match the requirements and return any which are
+    missing.
+    """
+    if not extras:
+        extras = tuple()
+
+    requires = get_distribution(package).requires(extras=extras)
+
+    missing_requirements = []
+    for requirement in requires:
+        try:
+            get_distribution(requirement)
+        except Exception:
+            missing_requirements.append(requirement.name)
+
+    return missing_requirements
+
+
+def missing_dependencies_by_extra(package="sunpy", exclude_extras=None):
+    """
+    Get all the specified extras for a package and report any missing dependencies.
+
+    This function will also return a "required" item in the dict which is the
+    dependencies associated with no extras.
+    """
+    exclude_extras = exclude_extras or []
+
+    distribution = get_distribution(package)
+    extras = distribution.extras
+
+    missing_dependencies = {"required": find_missing_dependencies(package)}
+    for extra in extras:
+        if extra in exclude_extras:
+            continue
+        missing_dependencies[extra] = find_missing_dependencies(package, (extra,))
+
+    return missing_dependencies
+
+
+def print_missing_dependencies_report(missing, package="sunpy"):
+    printed = False
+    required_missing = missing.pop("required")
+    if required_missing:
+        printed = True
+        print(f"The following packages are not installed but are required by {package}:")
+        for dep in required_missing:
+            print(f"* {dep}")
+
+    for extra_name, dependencies in missing.items():
+        if not dependencies:
+            continue
+
+        printed = True
+        print(
+            f"The following packages are not installed for the {package}[{extra_name}] requirement:")
+        for dep in dependencies:
+            print(f"  * {dep}")
+
+    return printed
+
+
+def _self_test_args(*, package=None, online=False, online_only=False, figure_only=False):
+    args = ["-W", "ignore"]
+    if online:
+        args.append("--remote-data=any")
+    if online_only:
+        args.append("--remote-data=any -m remote_data")
+    if package:
+        try:
+            importlib.import_module(f"sunpy.{package}")
+        except ModuleNotFoundError:
+            raise ModuleNotFoundError(f"sunpy.{package} was not found.")
+        args.extend(["--pyargs", f"sunpy.{package}"])
+    else:
+        args.extend(["--pyargs", "sunpy"])
+    if figure_only:
+        args.extend(["-m", "mpl_image_compare"])
+    return args
+
+
+def self_test(*, package=None, online=False, online_only=False, figure_only=False):
+    print("\n\n")
+    print("Starting sunpy self test...")
+    print("Checking for packages needed to run sunpy:")
+
+    missing = missing_dependencies_by_extra(exclude_extras=("asdf", "dask", "dev", "all", "docs"))
+    test_missing = missing.pop("tests")
+    printed = print_missing_dependencies_report(missing)
+
+    if not printed:
+        print("All required and optional sunpy dependencies are installed.")
+
+    if test_missing:
+        print("You do not have all the required dependencies installed to run the sunpy test suite.")
+        print("NOTE THIS JUST MEANS THE TESTS WILL NOT RUN, NOT THAT SUNPY MISCONFIGURED.")
+        print(
+            "If you want to run the sunpy tests install the 'tests' extra with `pip install sunpy[all,tests]`")
+        return
+
+    import pytest
+
+    print("Starting the sunpy test suite:")
+    print()
+    args = _self_test_args(package=package, online=online,
+                           online_only=online_only, figure_only=figure_only)
+    pytest.main(args)
diff --git a/sunpy/tests/tests/test_self_test.py b/sunpy/tests/tests/test_self_test.py
--- a/sunpy/tests/tests/test_self_test.py
+++ b/sunpy/tests/tests/test_self_test.py
@@ -1,30 +1,14 @@
-import os.path
 import warnings
-import importlib
 
 import pytest
 
 import sunpy
-import sunpy.tests.runner
+from sunpy.tests.self_test import _self_test_args
 from sunpy.util.exceptions import SunpyDeprecationWarning
 
-root_dir = os.path.dirname(os.path.abspath(sunpy.__file__))
-
-
-def test_import_runner():
-    """
-    When running the tests with setup.py, the test runner class is imported by
-    setup.py before coverage is watching.
-
-    To ensure that the coverage for sunpy/tests/runner.py is correctly
-    measured we force the interpreter to reload it here while coverage
-    is watching.
-    """
-    importlib.reload(sunpy.tests.runner)
-
 
 def test_main_nonexisting_module():
-    with pytest.raises(ValueError):
+    with pytest.raises(ModuleNotFoundError):
         sunpy.self_test(package='doesnotexist')
 
 
@@ -33,103 +17,43 @@ def test_main_stdlib_module():
     This test makes sure that the module is really searched within the sunpy
     package.
     """
-    with pytest.raises(ValueError):
+    with pytest.raises(ModuleNotFoundError):
         sunpy.self_test(package='random')
 
 
 def test_main_noargs(monkeypatch):
-    monkeypatch.setattr(pytest, 'main', lambda args, **kwargs: args)
-    args = sunpy.self_test()
-    assert args in (['sunpy', '-m', 'not figure'],
-                    [root_dir, '-m', 'not figure'])
+    test_args = _self_test_args()
+    assert test_args == ['-W', 'ignore', '--pyargs', 'sunpy']
 
 
 def test_main_submodule_map(monkeypatch):
-    monkeypatch.setattr(pytest, 'main', lambda args, **kwargs: args)
-    args = sunpy.self_test(package='map')
-    assert args in ([os.path.join('sunpy', 'map'), '-m', 'not figure'],
-                    [os.path.join(root_dir, 'map'), '-m', 'not figure'])
+    args = _self_test_args(package='map')
+    assert args == ['-W', 'ignore', '--pyargs', 'sunpy.map']
 
 
 def test_main_submodule_jsoc(monkeypatch):
-    monkeypatch.setattr(pytest, 'main', lambda args, **kwargs: args)
-    args = sunpy.self_test(package='net.jsoc')
-    assert args in ([os.path.join('sunpy', 'net', 'jsoc'), '-m', 'not figure'],
-                    [os.path.join(root_dir, 'net', 'jsoc'), '-m', 'not figure'])
+    args = _self_test_args(package='net.jsoc')
+    assert args == ['-W', 'ignore', '--pyargs', 'sunpy.net.jsoc']
 
 
 def test_main_exclude_remote_data(monkeypatch):
-    monkeypatch.setattr(pytest, 'main', lambda args, **kwargs: args)
-    args = sunpy.self_test(package='map', online=False)
-    assert args in ([os.path.join('sunpy', 'map'), '-m', 'not figure'],
-                    [os.path.join(root_dir, 'map'), '-m', 'not figure'])
+    args = _self_test_args(package='map', online=False)
+    assert args == ['-W', 'ignore', '--pyargs', 'sunpy.map']
 
 
 def test_main_include_remote_data(monkeypatch):
-    monkeypatch.setattr(pytest, 'main', lambda args, **kwargs: args)
-    args = sunpy.self_test(package='map', online=True)
-    assert args in ([os.path.join('sunpy', 'map'), '--remote-data=any', '-m', 'not figure'],
-                    [os.path.join(root_dir, 'map'), '--remote-data=any', '-m', 'not figure'])
+    args = _self_test_args(package='map', online=True)
+    assert args == ['-W', 'ignore', '--remote-data=any', '--pyargs', 'sunpy.map']
 
 
 def test_main_only_remote_data(monkeypatch):
-    monkeypatch.setattr(pytest, 'main', lambda args, **kwargs: args)
-    args = sunpy.self_test(package='map', online_only=True)
-    assert args in ([os.path.join('sunpy', 'map'), '-k remote_data', '--remote-data=any', '-m', 'not figure'],
-                    [os.path.join(root_dir, 'map'), '-k remote_data', '--remote-data=any', '-m', 'not figure'])
-
-
-def test_main_figures(monkeypatch):
-    monkeypatch.setattr(pytest, 'main', lambda args, **kwargs: args)
-    args = sunpy.self_test(figure=True)
-    assert args in (['sunpy'],
-                    [root_dir])
+    args = _self_test_args(package='map', online_only=True)
+    assert args == ['-W', 'ignore', '--remote-data=any -m remote_data', '--pyargs', 'sunpy.map']
 
 
 def test_main_figure_only(monkeypatch):
-    monkeypatch.setattr(pytest, 'main', lambda args, **kwargs: args)
-    args = sunpy.self_test(figure_only=True)
-    assert args in (['sunpy', '-m', 'figure'],
-                    [root_dir, '-m', 'figure'])
-
-
-def test_main_figure_dir(monkeypatch):
-    monkeypatch.setattr(pytest, 'main', lambda args, **kwargs: args)
-    args = sunpy.self_test(figure_only=True, figure_dir=".")
-    assert args in (['sunpy', '--figure_dir', '.', '-m', 'figure'],
-                    [root_dir, '--figure_dir', '.', '-m', 'figure'])
-
-
-def test_main_coverage(monkeypatch):
-    monkeypatch.setattr(pytest, 'main', lambda args, **kwargs: args)
-    args = sunpy.self_test(coverage=True)
-    for a in args:
-        assert a in [root_dir, 'sunpy', '--cov', '--cov-config', '-m',
-                     'not figure',
-                     os.path.join(root_dir, 'tests', 'coveragerc'),
-                     os.path.join('sunpy', 'tests', 'coveragerc')]
-
-
-def test_main_coverage_report(monkeypatch):
-    monkeypatch.setattr(pytest, 'main', lambda args, **kwargs: args)
-    args = sunpy.self_test(coverage=True, cov_report=True)
-    for a in args:
-        assert a in [root_dir, 'sunpy', '--cov', '--cov-config', '-m',
-                     'not figure',
-                     os.path.join(root_dir, 'tests', 'coveragerc'),
-                     os.path.join('sunpy', 'tests', 'coveragerc'),
-                     '--cov-report']
-
-
-def test_main_coverage_report_html(monkeypatch):
-    monkeypatch.setattr(pytest, 'main', lambda args, **kwargs: args)
-    args = sunpy.self_test(coverage=True, cov_report=True)
-    for a in args:
-        assert a in [root_dir, 'sunpy', '--cov', '--cov-config', '-m',
-                     'not figure',
-                     os.path.join(root_dir, 'tests', 'coveragerc'),
-                     os.path.join('sunpy', 'tests', 'coveragerc'),
-                     '--cov-report']
+    args = _self_test_args(figure_only=True)
+    assert args == ['-W', 'ignore', '--pyargs', 'sunpy', '-m', 'mpl_image_compare']
 
 
 def test_warnings():