Repository: sunpy/sunpy

Issue:
Should remote data manager namespace file names?
While working on sunpy/sunxspex#34 I thought, "what would happen if two affiliated packages both used the same name for a data file?" So I was wondering if we could do some mad introspection stuff to automatically namespace variables so it would be `"sunxspex.xray_abundancies"`?


Discussion:
I had thought about this during the initial development and thought I could use the function path + file name (`module.sub.funcname.file`) as the key in the data manager.

The method to get the function name of the callee from the called function turned out to be implementation dependent (cpython vs others), so I abandoned the idea.

Using just the module name might be easier(?).
What if they are the same file! Do we need then to download it twice?
IIRC, they won't download twice because we have a cache which handles the downloads for data manager. I'll have to check to confirm though.
I think as it's user facing name the module name + filename would be the best option.
If this feature is still required, can I take it up?
If yes, should I be namespacing it using the module name?
I don't see why not. 
This is turning out to be more complicated than I expected. Can someone help me with the following scenarios?

1. Suppose module `A` downloads a file and it gets namespaced as `A.test_file`. Later, module `B` also wants to use the same file. This time, `DataManager` will not download it again because `cache` is keeping track of downloads. While accessing this file  (using `manager.get()`) from `B`, how can we retrieve something that is namespaced with `A`? Would we need a different DB schema that can handle this?
2. Even if this happens, will integrity be maintained now that module `B` is using a file namespaced as `A.test_file`?


It's been a while since I looked at the "DataManager", is it not storing the id of the file (that is what get uses?) and then the filepath to get to the file?
It does, but not in the SQL storage - the id to path mapping is stored in a `dict` in memory.
I'll try modifying this `dict`, I think that should do.

Resolving patch:
diff --git a/sunpy/data/data_manager/cache.py b/sunpy/data/data_manager/cache.py
--- a/sunpy/data/data_manager/cache.py
+++ b/sunpy/data/data_manager/cache.py
@@ -38,7 +38,7 @@ def __init__(self, downloader, storage, cache_dir, expiry=10*u.day):
         self._cache_dir = Path(cache_dir)
         self._expiry = expiry if expiry is None else TimeDelta(expiry)
 
-    def download(self, urls, redownload=False):
+    def download(self, urls, namespace='', redownload=False):
         """
         Downloads the files from the urls.
 
@@ -83,7 +83,7 @@ def download(self, urls, redownload=False):
             else:
                 return Path(details['file_path'])
 
-        file_path, file_hash, url = self._download_and_hash(urls)
+        file_path, file_hash, url = self._download_and_hash(urls, namespace)
 
         self._storage.store({
             'file_hash': file_hash,
@@ -140,7 +140,7 @@ def _get_by_url(self, url):
         details = self._storage.find_by_key('url', url)
         return details
 
-    def _download_and_hash(self, urls):
+    def _download_and_hash(self, urls, namespace=''):
         """
         Downloads the file and returns the path, hash and url it used to download.
 
@@ -155,7 +155,7 @@ def _download_and_hash(self, urls):
             Path, hash and URL of the file.
         """
         def download(url):
-            path = self._cache_dir / get_filename(urlopen(url), url)
+            path = self._cache_dir / (namespace + get_filename(urlopen(url), url))
             # replacement_filename returns a string and we want a Path object
             path = Path(replacement_filename(str(path)))
             self._downloader.download(url, path)
diff --git a/sunpy/data/data_manager/manager.py b/sunpy/data/data_manager/manager.py
--- a/sunpy/data/data_manager/manager.py
+++ b/sunpy/data/data_manager/manager.py
@@ -26,6 +26,7 @@ def __init__(self, cache):
 
         self._file_cache = {}
 
+        self._namespace = None
         self._skip_hash_check = False
         self._skip_file: Dict[str, str] = {}
 
@@ -49,6 +50,7 @@ def require(self, name, urls, sha_hash):
         def decorator(func):
             @functools.wraps(func)
             def wrapper(*args, **kwargs):
+                self._namespace = self._get_module(func)
                 replace = self._skip_file.get(name, None)
                 if replace:
                     uri_parse = urlparse(replace['uri'])
@@ -61,14 +63,16 @@ def wrapper(*args, **kwargs):
                         file_path = uri_parse.netloc + uri_parse.path
                         file_hash = hash_file(file_path)
                     else:
-                        file_path, file_hash, _ = self._cache._download_and_hash([replace['uri']])
+                        file_path, file_hash, _ = self._cache._download_and_hash(
+                            [replace['uri']], self._namespace
+                        )
                     if replace['hash'] and file_hash != replace['hash']:
                         # if hash provided to replace function doesn't match the hash of the file
                         # raise error
                         raise ValueError(
                             "Hash provided to override_file does not match hash of the file.")
                 elif self._skip_hash_check:
-                    file_path = self._cache.download(urls, redownload=True)
+                    file_path = self._cache.download(urls, self._namespace, redownload=True)
                 else:
                     details = self._cache.get_by_hash(sha_hash)
                     if not details:
@@ -80,7 +84,7 @@ def wrapper(*args, **kwargs):
                             # in the database
                             raise ValueError(f"{urls} has already been downloaded, but no file "
                                              f"matching the hash {sha_hash} can be found.")
-                        file_path = self._cache.download(urls)
+                        file_path = self._cache.download(urls, self._namespace)
                         file_hash = hash_file(file_path)
                         if file_hash != sha_hash:
                             # the hash of the file downloaded does not match provided hash
@@ -96,7 +100,7 @@ def wrapper(*args, **kwargs):
                         if hash_file(details['file_path']) != details['file_hash']:
                             warnings.warn("Hashes do not match, the file will be redownloaded (could be be tampered/corrupted)",
                                           SunpyUserWarning)
-                            file_path = self._cache.download(urls, redownload=True)
+                            file_path = self._cache.download(urls, self._namespace, redownload=True)
                             # Recheck the hash again, if this fails, we will exit.
                             if hash_file(file_path) != details['file_hash']:
                                 raise RuntimeError("Redownloaded file also has the incorrect hash."
@@ -104,8 +108,12 @@ def wrapper(*args, **kwargs):
                         else:
                             file_path = details['file_path']
 
-                self._file_cache[name] = file_path
-                return func(*args, **kwargs)
+                if name not in self._file_cache:
+                    self._file_cache[name] = {}
+                self._file_cache[name][self._namespace] = file_path
+                result = func(*args, **kwargs)
+                self._namespace = None
+                return result
             return wrapper
 
         return decorator
@@ -172,10 +180,28 @@ def get(self, name):
         `KeyError`
             If ``name`` is not in the cache.
         """
-        return pathlib.Path(self._file_cache[name])
+        return pathlib.Path(self._file_cache[name][self._namespace])
 
     def _cache_has_file(self, urls):
         for url in urls:
             if self._cache._get_by_url(url):
                 return True
         return False
+
+    def _get_module(self, func):
+        """
+        Returns the name of the module (appended with a dot) that a function belongs to.
+
+        Parameters
+        ----------
+        func : function
+            A function whose module is to be found.
+
+        Returns
+        -------
+        A `str` that represents the module name appended with a dot.
+        """
+        module = func.__module__.lstrip('.').split('.')[0] + '.'
+        if module == '__main__.':
+            module = ''
+        return module


Test patch:
diff --git a/sunpy/data/data_manager/tests/conftest.py b/sunpy/data/data_manager/tests/conftest.py
--- a/sunpy/data/data_manager/tests/conftest.py
+++ b/sunpy/data/data_manager/tests/conftest.py
@@ -72,3 +72,27 @@ def foo(manager_tester=lambda x: 1):
         manager_tester(manager)
 
     return foo
+
+
+@pytest.fixture
+def module_patched_manager(tmp_path, downloader, storage, mocker):
+    def _get_module(func):
+        return 'fake_module.'
+
+    module_patched_manager = DataManager(Cache(downloader, storage, tmp_path))
+    module_patched_manager._tempdir = str(tmp_path)
+    module_patched_manager._get_module = _get_module
+    m = mock.Mock()
+    m.headers = {'Content-Disposition': 'test_file'}
+    mocker.patch('sunpy.data.data_manager.cache.urlopen', return_value=m)
+    yield module_patched_manager
+
+
+@pytest.fixture
+def data_function_from_fake_module(module_patched_manager):
+    @module_patched_manager.require(
+        'test_file', ['http://www.example.com/test_file'], mocks.MOCK_HASH)
+    def foo(manager_tester=lambda x: 1):
+        manager_tester(module_patched_manager)
+
+    return foo
diff --git a/sunpy/data/data_manager/tests/test_manager.py b/sunpy/data/data_manager/tests/test_manager.py
--- a/sunpy/data/data_manager/tests/test_manager.py
+++ b/sunpy/data/data_manager/tests/test_manager.py
@@ -12,7 +12,7 @@ def test_basic(storage, downloader, data_function):
 
     assert downloader.times_called == 1
     assert len(storage._store) == 1
-    assert Path(storage._store[0]['file_path']).name == ('test_file')
+    assert Path(storage._store[0]['file_path']).name == ('sunpy.test_file')
 
 
 def test_cache(manager, storage, downloader, data_function):
@@ -24,7 +24,7 @@ def test_cache(manager, storage, downloader, data_function):
 
     assert downloader.times_called == 1
     assert len(storage._store) == 1
-    assert Path(storage._store[0]['file_path']).name == ('test_file')
+    assert Path(storage._store[0]['file_path']).name == ('sunpy.test_file')
 
 
 def test_file_tampered(manager, storage, downloader, data_function):
@@ -32,13 +32,13 @@ def test_file_tampered(manager, storage, downloader, data_function):
     Test calling function multiple times does not redownload.
     """
     data_function()
-    write_to_test_file(manager._tempdir + '/test_file', 'b')
+    write_to_test_file(manager._tempdir + '/sunpy.test_file', 'b')
     with pytest.warns(SunpyUserWarning):
         data_function()
 
     assert downloader.times_called == 2
     assert len(storage._store) == 1
-    assert Path(storage._store[0]['file_path']).name == ('test_file')
+    assert Path(storage._store[0]['file_path']).name == ('sunpy.test_file')
 
 
 def test_wrong_hash_provided(manager):
@@ -60,7 +60,7 @@ def test_skip_all(manager, storage, downloader, data_function):
 
     assert downloader.times_called == 2
     assert len(storage._store) == 1
-    assert Path(storage._store[0]['file_path']).name == ('test_file')
+    assert Path(storage._store[0]['file_path']).name == ('sunpy.test_file')
 
 
 def test_override_file(manager, storage, downloader, data_function, tmpdir):
@@ -72,7 +72,7 @@ def default_tester(manager):
         """
         Function to test whether the file name is test_file.
         """
-        assert manager.get('test_file').name == ('test_file')
+        assert manager.get('test_file').name == ('sunpy.test_file')
 
     def override_file_tester(manager):
         """
@@ -160,3 +160,65 @@ def test_function():
 
     # SQLite should not throw an error
     test_function()
+
+
+def test_same_file_id_different_module(downloader, storage,
+                                       data_function, data_function_from_fake_module):
+    # Uses name 'test_file' to refer to the file
+    data_function()
+
+    # Change hash of the above file to allow MockDownloader to download another file
+    # Otherwise it will skip the download because a file with the same hash already exists
+    storage._store[0]['file_hash'] = 'abc'
+
+    # This function from a different module uses same name 'test_file' to refer to a different file
+    data_function_from_fake_module()
+
+    assert len(storage._store) == 2
+    assert downloader.times_called == 2
+
+    # Check if the files are namespaced correctly
+    assert Path(storage._store[0]['file_path']).name == 'sunpy.test_file'
+    assert Path(storage._store[1]['file_path']).name == 'fake_module.test_file'
+
+
+def test_namespacing_with_manager_override_file(module_patched_manager, downloader,
+                                                storage, data_function_from_fake_module):
+    # Download a file using manager.require()
+    data_function_from_fake_module()
+
+    assert len(storage._store) == 1
+    assert downloader.times_called == 1
+    assert Path(storage._store[0]['file_path']).name == 'fake_module.test_file'
+
+    # Override the file name with a different URI
+    with module_patched_manager.override_file(
+            'test_file', 'http://www.different_uri.com/new_file', MOCK_HASH):
+        data_function_from_fake_module()
+
+        assert downloader.times_called == 2
+
+        # New file entry is stored in manager._file_cache only
+        # It's not stored in InMemStorage or SqlStorage
+        assert len(storage._store) == 1
+
+        assert Path(
+            module_patched_manager._file_cache['test_file']['fake_module.']
+        ).name == 'fake_module.new_file'
+
+        # Storage still contains original test_file
+        assert Path(storage._store[0]['file_path']).name == 'fake_module.test_file'
+
+    # Request the original file again
+    data_function_from_fake_module()
+
+    # File dosn't get redownloaded, instead it is retrieved using the file hash
+    assert downloader.times_called == 2
+
+    # new_file entry in manager._file_cache is replaced with the original test_file
+    assert Path(
+        module_patched_manager._file_cache['test_file']['fake_module.']
+    ).name == 'fake_module.test_file'
+
+    # Storage still contains original test_file
+    assert Path(storage._store[0]['file_path']).name == 'fake_module.test_file'