Repository: fractal-analytics-platform/fractal-tasks-core (language: python)

Issue:
Explore using `pooch` for test datasets
### Current situation

1. In the future we may expand our test-data sources (e.g. by adding BIA #536). If so, we'll need reasonably flexible download procedures and it'd be best to avoid repeated downloads from any CI run.
2. Downloading data from Zenodo is currently based on custom functions, that requires us to know their API (with occasional problems like #597) and to use a bunch of requests/urrlib3 commands in our fixtures. These are mixed with actual fractal-tasks-core logic (e.g. [a workaround to update the ROI tables](https://github.com/fractal-analytics-platform/fractal-tasks-core/commit/f2bb44f4ceabe0c15c4e4449c522bb33c900c370)), which obviously does belong to our CI. It's not always easy to separate the two logical components (downloading data, and preparing them for our needs).

### Update proposal (not high priority)

An interesting option would be to use https://www.fatiando.org/pooch/latest/about.html. It abstracts the download of some datasets, and it stores them in a local cache, to avoid multiple downloads. Compared to our current approach, it also includes the checks to make sure the original files are there. In other words, it creates a local version of a bunch of datasets we may want to use for the CI or for examples.

Note that this would also play well with GitHub CI: we can make sure that the pooch folder is cached, so that downloads are never re-triggered (or very rarely).

Here are a few examples of how napari uses pooch in some examples: https://github.com/search?q=repo%3Anapari%2Fnapari+pooch+language%3APython&type=code&l=Python

And here is the same for scikit-image: https://github.com/search?q=repo%3Ascikit-image%2Fscikit-image+pooch+language%3APython&type=code&l=Python


Discussion / hints:
Sounds good! Recently heard the napari devs recommend pooch quite heavily in a napari workshop, so they seem to be happy with it 

Fix patch:
diff --git a/.github/workflows/ci_pip.yml b/.github/workflows/ci_pip.yml
index f4e41bc4f..e46369067 100644
--- a/.github/workflows/ci_pip.yml
+++ b/.github/workflows/ci_pip.yml
@@ -33,21 +33,10 @@ jobs:
         run: python -m pip install -e .
 
       - name: Install some testing dependencies (hard-coded)
-        run: python -m pip install pytest devtools jsonschema requests wget
-
-      - name: Cache Zenodo data
-        id: cache-zenodo-data
-        uses: actions/cache@v3
-        with:
-          path: tests/data/
-          key: zenodo-data
-
-      - name: Download Zenodo data
-        if: steps.cache-zenodo-data.outputs.cache-hit != 'true'
-        run: bash tests/data/download_zenodo_data.sh
+        run: python -m pip install pytest devtools jsonschema requests wget pooch
 
       - name: Test core library with pytest
-        run: pytest tests --ignore tests/tasks
+        run: python -m pytest tests --ignore tests/tasks -s --log-cli-level info
 
   tests_tasks:
     if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
@@ -74,18 +63,14 @@ jobs:
         run: python -m pip install -e .[fractal-tasks]
 
       - name: Install some testing dependencies (hard-coded)
-        run: python -m pip install pytest devtools jsonschema requests wget
+        run: python -m pip install pytest devtools jsonschema requests wget pooch
 
-      - name: Cache Zenodo data
-        id: cache-zenodo-data
+      - name: Cache Pooch folder
+        id: cache-pooch-folder
         uses: actions/cache@v3
         with:
-          path: tests/data/
-          key: zenodo-data
-
-      - name: Download Zenodo data
-        if: steps.cache-zenodo-data.outputs.cache-hit != 'true'
-        run: bash tests/data/download_zenodo_data.sh
+          path: ~/.cache/pooch
+          key: pooch-cache
 
       - name: Test tasks with pytest
-        run: pytest tests tests/tasks
+        run: python -m pytest tests tests/tasks -s --log-cli-level info
diff --git a/.github/workflows/ci_poetry.yml b/.github/workflows/ci_poetry.yml
index 4ae2f78d0..a3ef4fb4a 100644
--- a/.github/workflows/ci_poetry.yml
+++ b/.github/workflows/ci_poetry.yml
@@ -42,19 +42,8 @@ jobs:
       - name: Install dependencies (without extras)
         run: poetry install --with dev --without docs --no-interaction
 
-      - name: Cache Zenodo data
-        id: cache-zenodo-data
-        uses: actions/cache@v3
-        with:
-          path: tests/data/
-          key: zenodo-data
-
-      - name: Download Zenodo data
-        if: steps.cache-zenodo-data.outputs.cache-hit != 'true'
-        run: bash tests/data/download_zenodo_data.sh
-
       - name: Test core library with pytest
-        run: poetry run coverage run -m pytest tests --ignore tests/tasks
+        run: poetry run coverage run -m pytest tests --ignore tests/tasks -s --log-cli-level info
 
       - name: Upload coverage data
         uses: actions/upload-artifact@v3
@@ -99,19 +88,15 @@ jobs:
       - name: Check manifest task metadata
         run: poetry run python fractal_tasks_core/dev/check_manifest.py
 
-      - name: Cache Zenodo data
-        id: cache-zenodo-data
+      - name: Cache Pooch folder
+        id: cache-pooch-folder
         uses: actions/cache@v3
         with:
-          path: tests/data/
-          key: zenodo-data
-
-      - name: Download Zenodo data
-        if: steps.cache-zenodo-data.outputs.cache-hit != 'true'
-        run: bash tests/data/download_zenodo_data.sh
+          path: ~/.cache/pooch
+          key: pooch-cache
 
       - name: Test tasks with pytest
-        run: poetry run coverage run -m pytest tests/tasks
+        run: poetry run coverage run -m pytest tests/tasks -s --log-cli-level info
 
       - name: Upload coverage data
         uses: actions/upload-artifact@v3
diff --git a/CHANGELOG.md b/CHANGELOG.md
index d45a80ddc..c54345669 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,7 @@
 * Testing:
     * Align with new Zenodo API (\#601).
     * Update `test_valid_manifest` (\#606).
+    * Use [pooch](https://www.fatiando.org/pooch) to download test files (\#610).
 
 # 0.13.1
 
diff --git a/poetry.lock b/poetry.lock
index b2fde4a18..42c76a74e 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -3853,13 +3853,13 @@ testing = ["pytest", "pytest-benchmark"]
 
 [[package]]
 name = "pooch"
-version = "1.7.0"
+version = "1.8.0"
 description = "\"Pooch manages your Python library's sample data files: it automatically downloads and stores them in a local directory, with support for versioning and corruption checks.\""
-optional = true
+optional = false
 python-versions = ">=3.7"
 files = [
-    {file = "pooch-1.7.0-py3-none-any.whl", hash = "sha256:74258224fc33d58f53113cf955e8d51bf01386b91492927d0d1b6b341a765ad7"},
-    {file = "pooch-1.7.0.tar.gz", hash = "sha256:f174a1041b6447f0eef8860f76d17f60ed2f857dc0efa387a7f08228af05d998"},
+    {file = "pooch-1.8.0-py3-none-any.whl", hash = "sha256:1bfba436d9e2ad5199ccad3583cca8c241b8736b5bb23fe67c213d52650dbb66"},
+    {file = "pooch-1.8.0.tar.gz", hash = "sha256:f59981fd5b9b5d032dcde8f4a11eaa492c2ac6343fae3596a2fdae35fc54b0a0"},
 ]
 
 [package.dependencies]
@@ -5682,16 +5682,6 @@ docs = ["Sphinx (>=3.4)", "sphinx-rtd-theme (>=0.5)"]
 optional = ["python-socks", "wsaccel"]
 test = ["websockets"]
 
-[[package]]
-name = "wget"
-version = "3.2"
-description = "pure python download utility"
-optional = false
-python-versions = "*"
-files = [
-    {file = "wget-3.2.zip", hash = "sha256:35e630eca2aa50ce998b9b1a127bb26b30dfee573702782aa982f875e3f16061"},
-]
-
 [[package]]
 name = "wheel"
 version = "0.41.1"
@@ -5857,4 +5847,4 @@ fractal-tasks = ["Pillow", "cellpose", "imageio-ffmpeg", "napari-segment-blobs-a
 [metadata]
 lock-version = "2.0"
 python-versions = "^3.9"
-content-hash = "246e7f50c38dd454f5145cc383dec55790d5603aad3648b5a638338ba9cad0d4"
+content-hash = "6edcf502acdb856fb502c5772dbeab644710f1d63d1491b658378bcc23215858"
diff --git a/pyproject.toml b/pyproject.toml
index 3ee1873de..b4abc4084 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -62,8 +62,8 @@ pytest-pretty = "^1.1.0"
 jsonschema = "^4.16.0"
 mypy = "^1.3.0"
 requests = ">=2.28.0"
-wget = ">=3.2"
 napari-skimage-regionprops-mock = { file="tests/data/napari_workflows/mock_package/dist/napari_skimage_regionprops_mock-9.9.9-py3-none-any.whl" }
+pooch = "1.8.0"
 
 [tool.poetry.group.docs]
 optional = true


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
index d87c5a65f..dec6ebc5a 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -1,165 +1,9 @@
-import json
-import logging
-import os
-import shutil
-import time
 from pathlib import Path
 
-import anndata as ad
 import pytest
-import requests  # type: ignore
-import wget
-import zarr
-from devtools import debug
-
-from fractal_tasks_core.lib_regions_of_interest import reset_origin
-from fractal_tasks_core.lib_write import write_table
 
 
 @pytest.fixture(scope="session")
 def testdata_path() -> Path:
     TEST_DIR = Path(__file__).parent
     return TEST_DIR / "data/"
-
-
-@pytest.fixture(scope="session")
-def zenodo_images(testdata_path):
-    """
-    Inspired by
-    https://github.com/dvolgyes/zenodo_get/blob/master/zenodo_get/zget.py
-    """
-    t_start = time.perf_counter()
-
-    # Download images and metadata files
-    recordID = "7059515"
-    url = "10_5281_zenodo_7059515"
-    folder = str(testdata_path / f"10_5281_zenodo_{recordID}")
-    if os.path.isdir(folder):
-        print(f"{folder} already exists, skip download")
-    else:
-        os.makedirs(folder)
-        url = f"https://zenodo.org/api/records/{recordID}"
-        r = requests.get(url)
-        js = json.loads(r.text)
-        files = js["files"]
-        for f in files:
-            file_url = f["links"]["self"]
-            file_name = file_url.split("/")[-2]
-            wget.download(file_url, out=f"{folder}/{file_name}", bar=False)
-
-    # Add an image with invalid name, that should be skipped during parsing
-    with open(f"{folder}/invalid_path.png", "w") as f:
-        f.write("This file has an invalid filename, which cannot be parsed.")
-
-    t_end = time.perf_counter()
-    logging.warning(
-        f"\n    Time spent in zenodo_images: {t_end-t_start:.2f} s"
-    )
-
-    return folder
-
-
-@pytest.fixture(scope="session")
-def zenodo_images_multiplex(testdata_path, zenodo_images):
-    folder = str(testdata_path / "fake_multiplex")
-    cycle_folder_1 = str(Path(folder) / "cycle1")
-    cycle_folder_2 = str(Path(folder) / "cycle2")
-    cycle_folders = [cycle_folder_1, cycle_folder_2]
-    if os.path.isdir(folder):
-        print(f"{folder} already exists, skip zenodo_images_multiplex")
-    else:
-        os.makedirs(folder)
-        for cycle_folder in cycle_folders:
-            shutil.copytree(zenodo_images, cycle_folder)
-    return cycle_folders
-
-
-@pytest.fixture(scope="session")
-def zenodo_zarr(testdata_path, tmpdir_factory):
-    t_start = time.perf_counter()
-
-    doi = "10.5281/zenodo.8091756"
-    rootfolder = testdata_path / (doi.replace(".", "_").replace("/", "_"))
-    platenames = ["plate.zarr", "plate_mip.zarr"]
-    folders = [rootfolder / plate for plate in platenames]
-    zarrnames = [
-        "20200812-CardiomyocyteDifferentiation14-Cycle1.zarr",
-        "20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr",
-    ]
-
-    # Download dataset
-    if rootfolder.exists():
-        print(f"{str(rootfolder)} already exists, skip download part")
-    else:
-        rootfolder.mkdir()
-        tmp_path = tmpdir_factory.mktemp("zenodo_zarr")
-        for zarrname in zarrnames:
-            zipname = f"{zarrname}.zip"
-            url = f"https://zenodo.org/record/8091756/files/{zipname}"
-            debug(url)
-            wget.download(url, out=str(tmp_path / zipname), bar=None)
-            time.sleep(0.5)
-            shutil.unpack_archive(
-                str(tmp_path / zipname),
-                extract_dir=rootfolder,
-                format="zip",
-            )
-
-    # Based on the Zenodo OME-Zarrs, create the appropriate OME-Zarrs to be
-    # used in tests
-    for zarrname, folder in zip(zarrnames, folders):
-        if os.path.isdir(str(folder)):
-            shutil.rmtree(str(folder))
-        shutil.copytree(str(rootfolder / zarrname), str(folder))
-
-        # Update well/FOV ROI tables, by shifting their origin to 0
-        # TODO: remove this fix, by uploading new zarrs to zenodo (ref
-        # issue 526)
-        image_group_path = folder / "B/03/0"
-        group_image = zarr.open_group(str(image_group_path))
-        for table_name in ["FOV_ROI_table", "well_ROI_table"]:
-            table_path = str(image_group_path / "tables" / table_name)
-            old_table = ad.read_zarr(table_path)
-            new_table = reset_origin(old_table)
-            write_table(
-                group_image,
-                table_name,
-                new_table,
-                overwrite=True,
-                logger=logging.getLogger(),
-            )
-
-    folders = [str(f) for f in folders]
-
-    t_end = time.perf_counter()
-    logging.warning(f"\n    Time spent in zenodo_zarr: {t_end-t_start:.2f} s")
-    return folders
-
-
-@pytest.fixture(scope="function")
-def zenodo_zarr_metadata(testdata_path):
-    metadata_3D = {
-        "plate": ["plate.zarr"],
-        "well": ["plate.zarr/B/03"],
-        "image": ["plate.zarr/B/03/0/"],
-        "num_levels": 6,
-        "coarsening_xy": 2,
-        "original_paths": [str(testdata_path / "10_5281_zenodo_7059515/")],
-        "image_extension": "png",
-    }
-
-    metadata_2D = {
-        "plate": ["plate.zarr"],
-        "well": ["plate_mip.zarr/B/03/"],
-        "image": ["plate_mip.zarr/B/03/0/"],
-        "num_levels": 6,
-        "coarsening_xy": 2,
-        "original_paths": [str(testdata_path / "10_5281_zenodo_7059515/")],
-        "image_extension": "png",
-        "replicate_zarr": {
-            "suffix": "mip",
-            "sources": {"plate_mip": "/this/should/not/be/used/"},
-        },
-    }
-
-    return [metadata_3D, metadata_2D]
diff --git a/tests/data/download_zenodo_data.sh b/tests/data/download_zenodo_data.sh
deleted file mode 100644
index c95656b3e..000000000
--- a/tests/data/download_zenodo_data.sh
+++ /dev/null
@@ -1,39 +0,0 @@
-#!/bin/bash
-
-LIST_RECORD_ID="7059515 8091756"
-LIST_RECORD_ID="7059515 8091756"
-
-for RECORD_ID in $LIST_RECORD_ID; do
-    echo "****************************"
-    echo "START RECORD_ID=$RECORD_ID"
-    OUTPUT_FOLDER=tests/data/10_5281_zenodo_$RECORD_ID
-    echo "OUTPUT_FOLDER: $OUTPUT_FOLDER"
-
-    if [ -d $OUTPUT_FOLDER ]; then
-        echo "OUTPUT_FOLDER already exists. Exit."
-    else
-        mkdir $OUTPUT_FOLDER
-        FILES=`curl https://zenodo.org/api/records/$RECORD_ID | jq -r ".files[].links.self"`
-        echo "curl exit code: $?"
-        echo
-        for FILE in $FILES; do
-            FILEPATH=${FILE%"/content"}
-            FILENAME=`basename $FILEPATH`
-            echo "FILE:     $FILE"
-            echo "FILEPATH: $FILEPATH"
-            echo "FILENAME: $FILENAME"
-            echo
-            wget --no-verbose $FILE --output-document=${OUTPUT_FOLDER}/${FILENAME}
-            echo
-        done
-
-        if [ $RECORD_ID == "8091756" ]; then
-            unzip tests/data/10_5281_zenodo_8091756/20200812-CardiomyocyteDifferentiation14-Cycle1.zarr.zip -d tests/data/10_5281_zenodo_8091756
-            unzip tests/data/10_5281_zenodo_8091756/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr.zip -d tests/data/10_5281_zenodo_8091756
-        fi
-    fi
-
-    echo "END RECORD_ID=$RECORD_ID"
-    echo "****************************"
-    echo
-done
diff --git a/tests/tasks/__init__.py b/tests/tasks/__init__.py
index e69de29bb..988a631fb 100644
--- a/tests/tasks/__init__.py
+++ b/tests/tasks/__init__.py
@@ -0,0 +1,4 @@
+import pooch
+
+logger = pooch.get_logger()
+logger.setLevel("INFO")
diff --git a/tests/tasks/_validation.py b/tests/tasks/_validation.py
index 83fbc3fbe..8ed8b0840 100644
--- a/tests/tasks/_validation.py
+++ b/tests/tasks/_validation.py
@@ -1,12 +1,12 @@
 import glob
 import json
 import logging
-import urllib
 from pathlib import Path
 
 import anndata as ad
 import dask.array as da
 import numpy as np
+import pooch
 import zarr
 from devtools import debug
 from jsonschema import validate
@@ -19,13 +19,17 @@
 
 
 def validate_schema(*, path: str, type: str):
-    url: str = (
-        "https://raw.githubusercontent.com/ome/ngff/main/"
-        f"{__OME_NGFF_VERSION__}/schemas/{type}.schema"
+
+    file_path = pooch.retrieve(
+        url=(
+            "https://raw.githubusercontent.com/ome/ngff/main/"
+            f"{__OME_NGFF_VERSION__}/schemas/{type}.schema"
+        ),
+        known_hash=None,
     )
-    debug(url)
-    with urllib.request.urlopen(url) as fin:
-        schema: dict = json.load(fin)
+    debug(file_path)
+    with open(file_path) as fin:
+        schema = json.load(fin)
     debug(path)
     debug(type)
     with open(f"{path}/.zattrs", "r") as fin:
diff --git a/tests/tasks/conftest.py b/tests/tasks/conftest.py
index bbf4716dc..d3a3d00e1 100644
--- a/tests/tasks/conftest.py
+++ b/tests/tasks/conftest.py
@@ -1,9 +1,175 @@
 import logging
-import sys
+import os
+import shutil
 from pathlib import Path
 
-new_path = str(Path(__file__).parents[1])
-logging.info(f"Adding {new_path} to sys.path, and importing * from conftest")
-sys.path = [new_path] + sys.path
+import anndata as ad
+import pooch
+import pytest
+import zarr
+from devtools import debug
 
-from conftest import *  # noqa
+from ..conftest import *  # noqa
+from fractal_tasks_core.lib_regions_of_interest import reset_origin
+from fractal_tasks_core.lib_write import write_table
+
+
+@pytest.fixture(scope="session")
+def zenodo_images(testdata_path: Path) -> str:
+    """
+    1. Download image/metadata files from Zenodo;
+    2. Copy image/metadata files into a tests/data subfolder;
+    3. Add a spurious file.
+    """
+
+    DOI = "10.5281/zenodo.8287221"
+    DOI_slug = DOI.replace("/", "_").replace(".", "_")
+    rootfolder = testdata_path / DOI_slug
+    rootfolder.mkdir(exist_ok=True)
+
+    registry = {
+        "20200812-CardiomyocyteDifferentiation14-Cycle1_B03_T0001F001L01A01Z01C01.png": "md5:41c5d3612f166d30d694a6c9902a5839",  # noqa
+        "20200812-CardiomyocyteDifferentiation14-Cycle1_B03_T0001F001L01A01Z02C01.png": "md5:3aa92682cf731989cf4d3e0015f59ce0",  # noqa
+        "20200812-CardiomyocyteDifferentiation14-Cycle1_B03_T0001F002L01A01Z01C01.png": "md5:a3b0be2af486e08d1f009831d8656b80",  # noqa
+        "20200812-CardiomyocyteDifferentiation14-Cycle1_B03_T0001F002L01A01Z02C01.png": "md5:f1e0d50a1654ffd079504a036ff4a9e3",  # noqa
+        "MeasurementData.mlf": "md5:08898b37193727874b45c65a11754db9",
+        "MeasurementDetail.mrf": "md5:5fce4ca3e5ebc5f5be0b4945598e1ffb",
+    }
+    base_url = f"doi:{DOI}"
+    POOCH = pooch.create(
+        pooch.os_cache("pooch") / DOI_slug,
+        base_url,
+        registry=registry,
+        retry_if_failed=10,
+        allow_updates=False,
+    )
+
+    # Download files one by one, and copy them into rootfolder
+    debug(rootfolder)
+
+    for file_name in registry.keys():
+        debug(file_name)
+        file_path = POOCH.fetch(file_name)
+        shutil.copy(file_path, rootfolder / file_name)
+
+    # Add an image with invalid name, that should be skipped during parsing
+    with (rootfolder / "invalid_path.png").open("w") as f:
+        f.write("This file has an invalid filename, which cannot be parsed.")
+
+    return rootfolder.as_posix()
+
+
+@pytest.fixture(scope="session")
+def zenodo_images_multiplex(testdata_path: Path, zenodo_images: Path):
+    folder = str(testdata_path / "fake_multiplex")
+    cycle_folder_1 = str(Path(folder) / "cycle1")
+    cycle_folder_2 = str(Path(folder) / "cycle2")
+    cycle_folders = [cycle_folder_1, cycle_folder_2]
+    if os.path.isdir(folder):
+        print(f"{folder} already exists, skip zenodo_images_multiplex")
+    else:
+        os.makedirs(folder)
+        for cycle_folder in cycle_folders:
+            shutil.copytree(zenodo_images, cycle_folder)
+    return cycle_folders
+
+
+@pytest.fixture(scope="session")
+def zenodo_zarr(testdata_path: Path) -> list[str]:
+    """
+    This takes care of multiple steps:
+
+    1. Download/unzip two Zarr containers (3D and MIP) from Zenodo, via pooch
+    2. Copy the two Zarr containers into tests/data
+    3. Modify the Zarrs in tests/data, to add whatever is not in Zenodo
+    """
+
+    # 1 Download Zarrs from Zenodo
+    DOI = "10.5281/zenodo.8091756"
+    DOI_slug = DOI.replace("/", "_").replace(".", "_")
+    platenames = ["plate.zarr", "plate_mip.zarr"]
+    rootfolder = testdata_path / DOI_slug
+    folders = [rootfolder / plate for plate in platenames]
+
+    registry = {
+        "20200812-CardiomyocyteDifferentiation14-Cycle1.zarr.zip": "38b7894530f28fd6f55edf5272aaea104c11f36e28825446d23aff280f3a4290",  # noqa
+        "20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr.zip": "7efddc0bd20b186c28ca8373b1f7af2d1723d0663fe9438969dc79da02539175",  # noqa
+    }
+    base_url = f"doi:{DOI}"
+    POOCH = pooch.create(
+        pooch.os_cache("pooch") / DOI_slug,
+        base_url,
+        registry=registry,
+        retry_if_failed=10,
+        allow_updates=False,
+    )
+
+    for ind, file_name in enumerate(
+        [
+            "20200812-CardiomyocyteDifferentiation14-Cycle1.zarr",
+            "20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr",
+        ]
+    ):
+        # 1) Download/unzip a single Zarr from Zenodo
+        file_paths = POOCH.fetch(
+            f"{file_name}.zip", processor=pooch.Unzip(extract_dir=file_name)
+        )
+        zarr_full_path = file_paths[0].split(file_name)[0] + file_name
+        print(zarr_full_path)
+        folder = folders[ind]
+
+        # 2) Copy the downloaded Zarr into tests/data
+        if os.path.isdir(str(folder)):
+            shutil.rmtree(str(folder))
+        shutil.copytree(Path(zarr_full_path) / file_name, folder)
+
+        # 3) Modify the Zarr in tests/data to fit with expectations within
+        # fractal-tasks-core.
+        #
+        # Note: this currently consists in udating well/FOV ROI tables, by
+        # shifting their origin to 0.
+        # TODO: remove this fix, by uploading new zarrs to zenodo (ref 526).
+        image_group_path = folder / "B/03/0"
+        group_image = zarr.open_group(str(image_group_path))
+        for table_name in ["FOV_ROI_table", "well_ROI_table"]:
+            table_path = str(image_group_path / "tables" / table_name)
+            old_table = ad.read_zarr(table_path)
+            new_table = reset_origin(old_table)
+            write_table(
+                group_image,
+                table_name,
+                new_table,
+                overwrite=True,
+                logger=logging.getLogger(),
+            )
+
+    return [str(f) for f in folders]
+
+
+@pytest.fixture(scope="function")
+def zenodo_zarr_metadata(testdata_path: Path):
+    metadata_3D = {
+        "plate": ["plate.zarr"],
+        "well": ["plate.zarr/B/03"],
+        "image": ["plate.zarr/B/03/0/"],
+        "num_levels": 6,
+        "coarsening_xy": 2,
+        "original_paths": [str(testdata_path / "10_5281_zenodo_7059515/")],
+        "image_extension": "png",
+    }
+
+    metadata_2D = {
+        "plate": ["plate.zarr"],
+        "well": ["plate_mip.zarr/B/03/"],
+        "image": ["plate_mip.zarr/B/03/0/"],
+        "num_levels": 6,
+        "coarsening_xy": 2,
+        "original_paths": [str(testdata_path / "10_5281_zenodo_7059515/")],
+        "image_extension": "png",
+        "replicate_zarr": {
+            "suffix": "mip",
+            "sources": {"plate_mip": "/this/should/not/be/used/"},
+        },
+    }
+
+    return [metadata_3D, metadata_2D]
diff --git a/tests/test_unit_channels.py b/tests/test_unit_channels.py
index f3c9141c2..400d5f263 100644
--- a/tests/test_unit_channels.py
+++ b/tests/test_unit_channels.py
@@ -1,11 +1,14 @@
 import json
 from pathlib import Path
+from typing import Any
 
 import jsonschema
+import pooch
 import pytest
 import zarr
 from devtools import debug
 
+from fractal_tasks_core import __OME_NGFF_VERSION__
 from fractal_tasks_core.lib_channels import check_unique_wavelength_ids
 from fractal_tasks_core.lib_channels import check_well_channel_labels
 from fractal_tasks_core.lib_channels import define_omero_channels
@@ -124,20 +127,24 @@ def test_get_channel_from_list(testdata_path: Path):
 
 @pytest.fixture(scope="session")
 def omero_channel_schema():
-    import urllib.request
-    from fractal_tasks_core import __OME_NGFF_VERSION__
-
-    url = (
-        "https://raw.githubusercontent.com/ome/ngff/main/"
-        f"{__OME_NGFF_VERSION__}/schemas/image.schema"
+    file_path = pooch.retrieve(
+        url=(
+            "https://raw.githubusercontent.com/ome/ngff/main/"
+            f"{__OME_NGFF_VERSION__}/schemas/image.schema"
+        ),
+        known_hash=None,
     )
-    debug(url)
-    with urllib.request.urlopen(url) as fin:
+    debug(file_path)
+    with open(file_path) as fin:
         full_schema = json.load(fin)
+
     yield full_schema["properties"]["omero"]["properties"]["channels"]["items"]
 
 
-def test_define_omero_channels(testdata_path: Path, omero_channel_schema):
+def test_define_omero_channels(
+    testdata_path: Path,
+    omero_channel_schema: dict[str, Any],
+):
     """
     GIVEN a list of our custom `OmeroChannel` objects
     WHEN calling `define_omero_channels`