Repository: sunpy/sunpy

Issue:
Figure tests directory gets wiped after running via the test runner.
Just discovered that when I run:
```bash
python setup.py test --figure-only
```
The resulting figure images are saved in a tmp directory that gets wiped at the end of the process (this is under linux for me)

Using `pytest` adds the folder into my sunpy folder.




Discussion:
@dstansby fyi
Are you running that from a clean copy of the master branch? And do you get the same behaviour running `python setup.py test` (without the figure only flag)?
Yeah, so I think the test runner called via `setup.py` dumps the files into `/tmp/<some folder>` and once its done it cleans it up. While `pytest` seems to do the tests in place.

So looking at the old code, we used  to create a tmpdir directly using `tmpfile` and that folder was created outside of the normal tmp directory created by the test caller.  But what we how we want it to behave in each case is up for discussion. 
So it's possible to access the temporary test directory using `os.getcwd()`, but I have no idea to access the directory from which `python setup.py test` is run, to move the files there. Any ideas @Cadair? (yet again I am baffled by the whole `python setup.py test` business)
The only solution I can think of is to enable passing a base directory as a CLI argument to `py.test` because then the test command could just pass the correct dir to the pytest instance. I can not think of another way of letting pytest know where it should save the files.
So, the problem is that

- `python setup.py test` calls some code in `astropy_helpers`...
- ... which then calls some other code in `astropy`...
- ... which then creates a shell command, which is run using `subprocess` from a temp directory...
- ... so the original directory from which `setup.py` test was called from isn't visible to any tests that are run...
- ... which means it's basically impossible to extract the figures from the temp directory to the current working directory.

Previously this was got round by creating another temp dir, saving the figures to that, and just printing it to stdout after the tests. Options:

- We could revert to that, but I think it's a massive faff having to try and open that temp dir just to view the generated images.
- We could try and change something upstream (in `astropy`) to make extracting images from the tests easier
- We could drop `python setup.py test` in favor of `pytest` (which has downsides, but would be easiest for me to implement on the circleCI tests in the short term)

Thoughts?
@dstansby your code path analysis matches my understanding, the `setup.py test` command builds arguments for and then runs `sunpy.self_test` [here](https://github.com/sunpy/sunpy/blob/master/sunpy/tests/setup_command.py#L60).

So if we can add an optional parameter to the sunpy test runner that is the base directory for the figures i.e. defaults to `./` we can have `setup.py test` pass the base directory in which it was run to `sunpy.self_test`.
oh wait, I missed a step, `sunpy.self_test` calls `pytest.main` which means that we need to add a cli argument to pytest to make it all the way through the stack.
I think I am close to a solution on this.

Resolving patch:
diff --git a/sunpy/conftest.py b/sunpy/conftest.py
--- a/sunpy/conftest.py
+++ b/sunpy/conftest.py
@@ -13,8 +13,9 @@
 else:
     matplotlib.use('Agg')
 
+import sunpy.tests.helpers
 from sunpy.tests.hash import HASH_LIBRARY_NAME
-from sunpy.tests.helpers import new_hash_library, test_fig_dir
+from sunpy.tests.helpers import new_hash_library
 from sunpy.extern import six
 
 import pytest
@@ -35,6 +36,15 @@
     HAVE_REMOTEDATA = remotedata_spec is not None
 
 
+def pytest_addoption(parser):
+    parser.addoption("--figure_dir", action="store", default="./figure_test_images")
+
+
+@pytest.fixture(scope='session', autouse=True)
+def figure_base_dir(request):
+    sunpy.tests.helpers.figure_base_dir = request.config.getoption("--figure_dir")
+
+
 def pytest_runtest_setup(item):
     """
     pytest hook to skip all tests that have the mark 'online' if the
@@ -49,9 +59,10 @@ def pytest_runtest_setup(item):
 def pytest_unconfigure(config):
     if len(new_hash_library) > 0:
         # Write the new hash library in JSON
-        hashfile = os.path.join(test_fig_dir, HASH_LIBRARY_NAME)
+        figure_base_dir = os.path.abspath(config.getoption("--figure_dir"))
+        hashfile = os.path.join(figure_base_dir, HASH_LIBRARY_NAME)
         with open(hashfile, 'w') as outfile:
             json.dump(new_hash_library, outfile, sort_keys=True, indent=4, separators=(',', ': '))
 
-        print('All images from image tests can be found in {0}'.format(test_fig_dir))
+        print('All images from image tests can be found in {0}'.format(figure_base_dir))
         print("The corresponding hash library is {0}".format(hashfile))


Test patch:
diff --git a/sunpy/tests/helpers.py b/sunpy/tests/helpers.py
--- a/sunpy/tests/helpers.py
+++ b/sunpy/tests/helpers.py
@@ -2,18 +2,14 @@
 # Lovingly borrowed from Astropy
 # Licensed under a 3-clause BSD style license - see licences/ASTROPY.rst
 
-from __future__ import absolute_import, division, print_function
-import warnings
-import tempfile
-import platform
 import os
 import datetime
+import platform
+import warnings
 
 import pytest
-import numpy as np
 import matplotlib.pyplot as plt
 
-import astropy.units as u
 from astropy.utils.decorators import wraps
 
 from sunpy.tests import hash
@@ -35,7 +31,7 @@
 
 try:
     from sunpy.io import _pyana
-except ImportError as e:
+except ImportError:
     SKIP_ANA = True
 else:
     SKIP_ANA = False
@@ -52,9 +48,8 @@ def warnings_as_errors(request):
     request.addfinalizer(lambda *args: warnings.resetwarnings())
 
 
-hash_library = hash.hash_library
 new_hash_library = {}
-test_fig_dir = 'result_images/{:%H%M%S}'.format(datetime.datetime.now())
+figure_base_dir = None
 
 
 def figure_test(test_function):
@@ -78,6 +73,8 @@ def test_simple_plot():
     def wrapper(*args, **kwargs):
         if not os.path.exists(hash.HASH_LIBRARY_FILE):
             pytest.xfail('Could not find a figure hash library at {}'.format(hash.HASH_LIBRARY_FILE))
+        if figure_base_dir is None:
+            pytest.xfail("No directory to save figures to found")
 
         name = "{0}.{1}".format(test_function.__module__,
                                 test_function.__name__)
@@ -88,9 +85,9 @@ def wrapper(*args, **kwargs):
             fig = plt.gcf()
 
         # Save the image that was generated
-        if not os.path.exists(test_fig_dir):
-            os.makedirs(test_fig_dir)
-        result_image_loc = os.path.join(test_fig_dir, '{}.png'.format(name))
+        if not os.path.exists(figure_base_dir):
+            os.makedirs(figure_base_dir)
+        result_image_loc = os.path.join(figure_base_dir, '{}.png'.format(name))
         plt.savefig(result_image_loc)
         plt.close()
 
@@ -100,10 +97,10 @@ def wrapper(*args, **kwargs):
         imgdata.close()
 
         new_hash_library[name] = figure_hash
-        if name not in hash_library:
+        if name not in hash.hash_library:
             pytest.fail("Hash not present: {0}".format(name))
 
-        if hash_library[name] != figure_hash:
+        if hash.hash_library[name] != figure_hash:
             raise RuntimeError('Figure hash does not match expected hash.\n'
                                'New image generated and placed at {}'.format(result_image_loc))
 
diff --git a/sunpy/tests/runner.py b/sunpy/tests/runner.py
--- a/sunpy/tests/runner.py
+++ b/sunpy/tests/runner.py
@@ -72,6 +72,17 @@ def figure_only(self, figure, kwargs):
 
         return []
 
+    @keyword()
+    def figure_dir(self, figure_dir, kwargs):
+        """
+        figure_tests : str, optional
+            Set the output directory for figure test images and hashes.
+        """
+        if figure_dir:
+            return ['--figure_dir', figure_dir]
+
+        return []
+
     # Define this to change the default value to None
     @keyword()
     def plugins(self, plugins, kwargs):
diff --git a/sunpy/tests/setup_command.py b/sunpy/tests/setup_command.py
--- a/sunpy/tests/setup_command.py
+++ b/sunpy/tests/setup_command.py
@@ -7,8 +7,7 @@
 This file is designed to be imported and ran only via setup.py, hence it's
 dependency on astropy_helpers which will be available in that context.
 """
-from __future__ import absolute_import, division, print_function
-
+import os
 import copy
 
 from astropy_helpers.commands.test import AstropyTest
@@ -69,6 +68,7 @@ def generate_testing_command(self):
                'online_only={1.online_only!r}, '
                'figure={1.figure!r}, '
                'figure_only={1.figure_only!r}, '
+               'figure_dir="{figure_dir}", '
                'pep8={1.pep8!r}, '
                'pdb={1.pdb!r}, '
                'open_files={1.open_files!r}, '
@@ -80,5 +80,6 @@ def generate_testing_command(self):
                'sys.exit(result)')
         return cmd.format('pass',
                           self,
+                          figure_dir=os.path.join(os.path.abspath('.'), "figure_test_images"),
                           cmd_pre=cmd_pre,
                           cmd_post=cmd_post)