Repository: fatiando/pooch

Issue:
Populate registry via data repository API
**Description of the desired feature:**

When implementing #318, I realized that the DOI support in pooch currently does not allow to automatically populate the pooch's registry using the API, although the information (incl. checksums) is readily available (checked DataVerse and Figshare, Zenodo should have the same). I would like to add this at least for DataVerse, as only with this feature, the built-in pooch solution is superior to my [prior implementation](https://github.com/ssciwr/dataverse_pooch). It would make sense however to do this in a unified way across different data repositories.

*Proposal*: Add a function `registry_from_doi` to that performs a dispatch based on the data repository type to implementation functions that map a DOI to a registry dictionary. Sample code:

```python
MYPOOCH = pooch.create(base_url="doi:..", ...)
pooch.registry_from_doi(MYPOOCH)
```
There are many ways to formulate this interface. I would prefer one that does not need redundant specification of the DOI (so it has the pooch object). Alternatively, a special marker object could be passed to `pooch.create`'s `registry=` argument.

The above proposal could be combined with a refactoring that abstracts data repositories into objects with a well-defined interface, instead of juggling around with functions. This is a matter of personal taste though.

**Are you willing to help implement and maintain this feature?** 

Yes, I can implement it and will be around for helping with maintenance.

Discussion:
dokempf: @leouieda This is now ready for review.
dokempf: @leouieda Sorry for the ping on this, but I have my conference presentation coming up on March 2nd where it would like to present this feature - ideally as part of a released version of pooch.
santisoler: Hi @dokempf! Sounds awesome you want to present this feature in a conference. Sorry for keep you on hold with this PR, I'll try to review it this Friday so we can merge it.

We had a few patches on Pooch in the last month, so I think we can make a patch or a minor release really soon. Ping us if we forget (we are just too busy 😭 ).
dokempf: @santisoler All comments addressed, thanks a lot for the review.
santisoler: Thanks @dokempf for addressing my comments. This is looking great. I just pushed a few changes, mostly improving some docstrings and adding a small paragraph to the user guide instructing how to use this new function when working with DOIs. I'm merging this 🚀 

I'm planning to make a Pooch release before the end of the month. There are one or two bugs I would like to solve before that, but I'm committing myself to get it ready before your presentation at the conference.


Patch:
diff --git a/doc/protocols.rst b/doc/protocols.rst
index e0ca431..94fe456 100644
--- a/doc/protocols.rst
+++ b/doc/protocols.rst
@@ -103,3 +103,24 @@ figshare dataset:
     ``doi:10.6084/m9.figshare.c.4362224.v1``. Attempting to download files
     from a figshare collection will raise an error.
     See `issue #274 <https://github.com/fatiando/pooch/issues/274>`__ details.
+
+Since this type of repositories store information about the files contained in
+them, we can avoid having to manually type the registry with the file names and
+their hashes.
+Instead, we can use the :meth:`pooch.Pooch.load_registry_from_doi` to
+automatically populate the registry:
+
+.. code-block:: python
+
+    POOCH = pooch.create(
+        path=pooch.os_cache("plumbus"),
+        # Use the figshare DOI
+        base_url="doi:10.6084/m9.figshare.14763051.v1/",
+        registry=None,
+    )
+
+    # Automatically populate the registry
+    POOCH.load_registry_from_doi()
+
+    # Fetch one of the files in the repository
+    fname = POOCH.fetch("tiny-data.txt")
diff --git a/pooch/core.py b/pooch/core.py
index 9375482..aa8ab13 100644
--- a/pooch/core.py
+++ b/pooch/core.py
@@ -27,7 +27,7 @@ from .utils import (
     os_cache,
     unique_file_name,
 )
-from .downloaders import choose_downloader
+from .downloaders import DOIDownloader, choose_downloader, doi_to_repository
 
 
 def retrieve(
@@ -670,6 +670,36 @@ class Pooch:
                         self.urls[file_name] = file_url
                     self.registry[file_name] = file_checksum.lower()
 
+    def load_registry_from_doi(self):
+        """
+        Populate the registry using the data repository API
+
+        Fill the registry with all the files available in the data repository,
+        along with their hashes. It will make a request to the data repository
+        API to retrieve this information. No file is downloaded during this
+        process.
+
+        .. important::
+
+            This method is intended to be used only when the ``base_url`` is
+            a DOI.
+        """
+
+        # Ensure that this is indeed a DOI-based pooch
+        downloader = choose_downloader(self.base_url)
+        if not isinstance(downloader, DOIDownloader):
+            raise ValueError(
+                f"Invalid base_url '{self.base_url}': "
+                + "Pooch.load_registry_from_doi is only implemented for DOIs"
+            )
+
+        # Create a repository instance
+        doi = self.base_url.replace("doi:", "")
+        repository = doi_to_repository(doi)
+
+        # Call registry population for this repository
+        return repository.populate_registry(self)
+
     def is_available(self, fname, downloader=None):
         """
         Check availability of a remote file without downloading it.
diff --git a/pooch/downloaders.py b/pooch/downloaders.py
index b4ff0a0..eabb08a 100644
--- a/pooch/downloaders.py
+++ b/pooch/downloaders.py
@@ -593,33 +593,8 @@ class DOIDownloader:  # pylint: disable=too-few-public-methods
 
         """
 
-        repositories = [
-            FigshareRepository,
-            ZenodoRepository,
-            DataverseRepository,
-        ]
-
-        # Extract the DOI and the repository information
         parsed_url = parse_url(url)
-        doi = parsed_url["netloc"]
-        archive_url = doi_to_url(doi)
-
-        # Try the converters one by one until one of them returned a URL
-        data_repository = None
-        for repo in repositories:
-            if data_repository is None:
-                data_repository = repo.initialize(
-                    archive_url=archive_url,
-                    doi=doi,
-                )
-
-        if data_repository is None:
-            repository = parse_url(archive_url)["netloc"]
-            raise ValueError(
-                f"Invalid data repository '{repository}'. "
-                "To request or contribute support for this repository, "
-                "please open an issue at https://github.com/fatiando/pooch/issues"
-            )
+        data_repository = doi_to_repository(parsed_url["netloc"])
 
         # Resolve the URL
         file_name = parsed_url["path"].split("/")[-1]
@@ -657,6 +632,59 @@ def doi_to_url(doi):
     return url
 
 
+def doi_to_repository(doi):
+    """
+    Instantiate a data repository instance from a given DOI.
+
+    This function implements the chain of responsibility dispatch
+    to the correct data repository class.
+
+    Parameters
+    ----------
+    doi : str
+        The DOI of the archive.
+
+    Returns
+    -------
+    data_repository : DataRepository
+        The data repository object
+    """
+
+    # This should go away in a separate issue: DOI handling should
+    # not rely on the (non-)existence of trailing slashes. The issue
+    # is documented in https://github.com/fatiando/pooch/issues/324
+    if doi[-1] == "/":
+        doi = doi[:-1]
+
+    repositories = [
+        FigshareRepository,
+        ZenodoRepository,
+        DataverseRepository,
+    ]
+
+    # Extract the DOI and the repository information
+    archive_url = doi_to_url(doi)
+
+    # Try the converters one by one until one of them returned a URL
+    data_repository = None
+    for repo in repositories:
+        if data_repository is None:
+            data_repository = repo.initialize(
+                archive_url=archive_url,
+                doi=doi,
+            )
+
+    if data_repository is None:
+        repository = parse_url(archive_url)["netloc"]
+        raise ValueError(
+            f"Invalid data repository '{repository}'. "
+            "To request or contribute support for this repository, "
+            "please open an issue at https://github.com/fatiando/pooch/issues"
+        )
+
+    return data_repository
+
+
 class DataRepository:  # pylint: disable=too-few-public-methods, missing-class-docstring
     @classmethod
     def initialize(cls, doi, archive_url):  # pylint: disable=unused-argument
@@ -697,11 +725,24 @@ class DataRepository:  # pylint: disable=too-few-public-methods, missing-class-d
 
         raise NotImplementedError  # pragma: no cover
 
+    def populate_registry(self, pooch):
+        """
+        Populate the registry using the data repository's API
+
+        Parameters
+        ----------
+        pooch : Pooch
+            The pooch instance that the registry will be added to.
+        """
+
+        raise NotImplementedError  # pragma: no cover
+
 
 class ZenodoRepository(DataRepository):  # pylint: disable=missing-class-docstring
     def __init__(self, doi, archive_url):
         self.archive_url = archive_url
         self.doi = doi
+        self._api_response = None
 
     @classmethod
     def initialize(cls, doi, archive_url):
@@ -729,6 +770,18 @@ class ZenodoRepository(DataRepository):  # pylint: disable=missing-class-docstri
 
         return cls(doi, archive_url)
 
+    @property
+    def api_response(self):
+        """Cached API response from Zenodo"""
+
+        if self._api_response is None:
+            article_id = self.archive_url.split("/")[-1]
+            self._api_response = requests.get(
+                f"https://zenodo.org/api/records/{article_id}"
+            ).json()
+
+        return self._api_response
+
     def download_url(self, file_name):
         """
         Use the repository API to get the download URL for a file given
@@ -744,10 +797,8 @@ class ZenodoRepository(DataRepository):  # pylint: disable=missing-class-docstri
         download_url : str
             The HTTP URL that can be used to download the file.
         """
-        article_id = self.archive_url.split("/")[-1]
-        # With the ID, we can get a list of files and their download links
-        article = requests.get(f"https://zenodo.org/api/records/{article_id}").json()
-        files = {item["key"]: item for item in article["files"]}
+
+        files = {item["key"]: item for item in self.api_response["files"]}
         if file_name not in files:
             raise ValueError(
                 f"File '{file_name}' not found in data archive {self.archive_url} (doi:{self.doi})."
@@ -755,11 +806,25 @@ class ZenodoRepository(DataRepository):  # pylint: disable=missing-class-docstri
         download_url = files[file_name]["links"]["self"]
         return download_url
 
+    def populate_registry(self, pooch):
+        """
+        Populate the registry using the data repository's API
+
+        Parameters
+        ----------
+        pooch : Pooch
+            The pooch instance that the registry will be added to.
+        """
+
+        for filedata in self.api_response["files"]:
+            pooch.registry[filedata["key"]] = filedata["checksum"]
+
 
 class FigshareRepository(DataRepository):  # pylint: disable=missing-class-docstring
     def __init__(self, doi, archive_url):
         self.archive_url = archive_url
         self.doi = doi
+        self._api_response = None
 
     @classmethod
     def initialize(cls, doi, archive_url):
@@ -787,6 +852,25 @@ class FigshareRepository(DataRepository):  # pylint: disable=missing-class-docst
 
         return cls(doi, archive_url)
 
+    @property
+    def api_response(self):
+        """Cached API response from Figshare"""
+
+        if self._api_response is None:
+            # Use the figshare API to find the article ID from the DOI
+            article = requests.get(
+                f"https://api.figshare.com/v2/articles?doi={self.doi}"
+            ).json()[0]
+            article_id = article["id"]
+            # With the ID, we can get a list of files and their download links
+            response = requests.get(
+                f"https://api.figshare.com/v2/articles/{article_id}/files"
+            )
+            response.raise_for_status()
+            self._api_response = response.json()
+
+        return self._api_response
+
     def download_url(self, file_name):
         """
         Use the repository API to get the download URL for a file given
@@ -803,17 +887,7 @@ class FigshareRepository(DataRepository):  # pylint: disable=missing-class-docst
             The HTTP URL that can be used to download the file.
         """
 
-        # Use the figshare API to find the article ID from the DOI
-        article = requests.get(
-            f"https://api.figshare.com/v2/articles?doi={self.doi}"
-        ).json()[0]
-        article_id = article["id"]
-        # With the ID, we can get a list of files and their download links
-        response = requests.get(
-            f"https://api.figshare.com/v2/articles/{article_id}/files"
-        )
-        response.raise_for_status()
-        files = {item["name"]: item for item in response.json()}
+        files = {item["name"]: item for item in self.api_response}
         if file_name not in files:
             raise ValueError(
                 f"File '{file_name}' not found in data archive {self.archive_url} (doi:{self.doi})."
@@ -821,11 +895,25 @@ class FigshareRepository(DataRepository):  # pylint: disable=missing-class-docst
         download_url = files[file_name]["download_url"]
         return download_url
 
+    def populate_registry(self, pooch):
+        """
+        Populate the registry using the data repository's API
+
+        Parameters
+        ----------
+        pooch : Pooch
+            The pooch instance that the registry will be added to.
+        """
+
+        for filedata in self.api_response:
+            pooch.registry[filedata["name"]] = f"md5:{filedata['computed_md5']}"
+
 
 class DataverseRepository(DataRepository):  # pylint: disable=missing-class-docstring
     def __init__(self, doi, archive_url):
         self.archive_url = archive_url
         self.doi = doi
+        self._api_response = None
 
     @classmethod
     def initialize(cls, doi, archive_url):
@@ -847,17 +935,48 @@ class DataverseRepository(DataRepository):  # pylint: disable=missing-class-docs
         """
 
         # Access the DOI as if this was a DataVerse instance
+        response = cls._get_api_response(doi, archive_url)
+
+        # If we failed, this is probably not a DataVerse instance
+        if 400 <= response.status_code < 600:
+            return None
+
+        # Initialize the repository and overwrite the api response
+        repository = cls(doi, archive_url)
+        repository.api_response = response
+        return repository
+
+    @classmethod
+    def _get_api_response(cls, doi, archive_url):
+        """
+        Perform the actual API request
+
+        This has been separated into a separate ``classmethod``, as it can be
+        used prior and after the initialization.
+        """
         parsed = parse_url(archive_url)
         response = requests.get(
             f"{parsed['protocol']}://{parsed['netloc']}/api/datasets/"
             f":persistentId?persistentId=doi:{doi}"
         )
+        return response
 
-        # If we failed, this is probably not a DataVerse instance
-        if 400 <= response.status_code < 600:
-            return None
+    @property
+    def api_response(self):
+        """Cached API response from a DataVerse instance"""
 
-        return cls(doi, archive_url)
+        if self._api_response is None:
+            self._api_response = self._get_api_response(
+                self.doi, self.archive_url
+            )  # pragma: no cover
+
+        return self._api_response
+
+    @api_response.setter
+    def api_response(self, response):
+        """Update the cached API response"""
+
+        self._api_response = response
 
     def download_url(self, file_name):
         """
@@ -875,15 +994,10 @@ class DataverseRepository(DataRepository):  # pylint: disable=missing-class-docs
             The HTTP URL that can be used to download the file.
         """
 
-        # Access the DOI as if this was a DataVerse instance
         parsed = parse_url(self.archive_url)
-        response = requests.get(
-            f"{parsed['protocol']}://{parsed['netloc']}/api/datasets/"
-            f":persistentId?persistentId=doi:{self.doi}"
-        )
 
         # Iterate over the given files until we find one of the requested name
-        for filedata in response.json()["data"]["latestVersion"]["files"]:
+        for filedata in self.api_response.json()["data"]["latestVersion"]["files"]:
             if file_name == filedata["dataFile"]["filename"]:
                 return (
                     f"{parsed['protocol']}://{parsed['netloc']}/api/access/datafile/"
@@ -893,3 +1007,18 @@ class DataverseRepository(DataRepository):  # pylint: disable=missing-class-docs
         raise ValueError(
             f"File '{file_name}' not found in data archive {self.archive_url} (doi:{self.doi})."
         )
+
+    def populate_registry(self, pooch):
+        """
+        Populate the registry using the data repository's API
+
+        Parameters
+        ----------
+        pooch : Pooch
+            The pooch instance that the registry will be added to.
+        """
+
+        for filedata in self.api_response.json()["data"]["latestVersion"]["files"]:
+            pooch.registry[
+                filedata["dataFile"]["filename"]
+            ] = f"md5:{filedata['dataFile']['md5']}"


Test patch:
diff --git a/pooch/tests/test_core.py b/pooch/tests/test_core.py
index addf0fa..bb5ec85 100644
--- a/pooch/tests/test_core.py
+++ b/pooch/tests/test_core.py
@@ -28,6 +28,7 @@ from .utils import (
     data_over_ftp,
     pooch_test_figshare_url,
     pooch_test_zenodo_url,
+    pooch_test_dataverse_url,
     pooch_test_registry,
     check_tiny_data,
     check_large_data,
@@ -40,6 +41,7 @@ REGISTRY = pooch_test_registry()
 BASEURL = pooch_test_url()
 FIGSHAREURL = pooch_test_figshare_url()
 ZENODOURL = pooch_test_zenodo_url()
+DATAVERSEURL = pooch_test_dataverse_url()
 REGISTRY_CORRUPTED = {
     # The same data file but I changed the hash manually to a wrong one
     "tiny-data.txt": "098h0894dba14b12085eacb204284b97e362f4f3e5a5807693cc90ef415c1b2d"
@@ -135,7 +137,9 @@ def test_pooch_local(data_dir_mirror):
 
 @pytest.mark.network
 @pytest.mark.parametrize(
-    "url", [BASEURL, FIGSHAREURL, ZENODOURL], ids=["https", "figshare", "zenodo"]
+    "url",
+    [BASEURL, FIGSHAREURL, ZENODOURL, DATAVERSEURL],
+    ids=["https", "figshare", "zenodo", "dataverse"],
 )
 def test_pooch_custom_url(url):
     "Have pooch download the file from URL that is not base_url"
@@ -159,7 +163,9 @@ def test_pooch_custom_url(url):
 
 @pytest.mark.network
 @pytest.mark.parametrize(
-    "url", [BASEURL, FIGSHAREURL, ZENODOURL], ids=["https", "figshare", "zenodo"]
+    "url",
+    [BASEURL, FIGSHAREURL, ZENODOURL, DATAVERSEURL],
+    ids=["https", "figshare", "zenodo", "dataverse"],
 )
 def test_pooch_download(url):
     "Setup a pooch that has no local data and needs to download"
@@ -608,3 +614,36 @@ def test_stream_download(fname):
         stream_download(url, destination, known_hash, downloader, pooch=None)
         assert destination.exists()
         check_tiny_data(str(destination))
+
+
+@pytest.mark.parametrize(
+    "url",
+    [FIGSHAREURL, ZENODOURL, DATAVERSEURL],
+    ids=["figshare", "zenodo", "dataverse"],
+)
+def test_load_registry_from_doi(url):
+    """Check that the registry is correctly populated from the API"""
+    with TemporaryDirectory() as local_store:
+        path = os.path.abspath(local_store)
+        pup = Pooch(path=path, base_url=url)
+        pup.load_registry_from_doi()
+
+        # Check the existence of all files in the registry
+        assert len(pup.registry) == 2
+        assert "tiny-data.txt" in pup.registry
+        assert "store.zip" in pup.registry
+
+        # Ensure that all files have correct checksums by fetching them
+        for filename in pup.registry:
+            pup.fetch(filename)
+
+
+def test_wrong_load_registry_from_doi():
+    """Check that non-DOI URLs produce an error"""
+
+    pup = Pooch(path="", base_url=BASEURL)
+
+    with pytest.raises(ValueError) as exc:
+        pup.load_registry_from_doi()
+
+    assert "only implemented for DOIs" in str(exc.value)