Repository: NCAS-CMS/PyActiveStorage
Pull request #245: Enable https reduction (off NGINX server only) and auto-detect `storage_type`
Author: login: valeriupredoi
Labels: name: enhancement
description: New feature or request
Created: 2025-03-10T16:05:45Z | Merged: 2025-03-20T17:23:55Z

<!--
    Thank you for contributing to our project!

    Please do not delete this text completely, but read the text below and keep
    items that seem relevant. If in doubt, just keep everything and add your
    own text at the top, a reviewer will update the checklist for you.

-->

## Description

<!--
    Please describe your changes here, especially focusing on why this pull
    request makes PyActiveStorage better and what problem it solves.

    Please fill in the GitHub issue that is closed by this pull request,
    e.g. Closes #1903
-->

Yet another contribution towards #231 

- introduce a utility that should be able to detect storage type, and enable the user not to be forced to specify the `storage_type` kwarg in `Active`
- allow for files and datasets from files on https servers to be handled, with the caveat that reductions happen inside the PyAS client, and not yet delegated to Reductionist
- testing with https files is done with actual data off the CEDA CMIP6 thredds archive so those tests register quite a bit of test times variability 0 it all depends on how busy the server is, yesterday for instance, they were taking order 3min, usually they should take at most 15-20s if the server is not busy; I'd like to keep them and not `skip` them since they are proper real world used cases

Reduction inside PyAS client is obviously slower than if delegated to Reductionist, but it's still a good exercise, and it scales very well (ie 2x data = 2x time)

yay we got Codecov back online too! [Report](https://app.codecov.io/gh/NCAS-CMS/PyActiveStorage/pull/245?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAS-CMS) complains about 4 lines I can't really test unless I spool up a Minio client, which is done via the S3 test, but by the main's not-S3 tests, so that's covered too :beer: 


## Before you get started

- [x] [☝ Create an issue](https://github.com/valeriupredoi/PyActiveStorage/issues) to discuss what you are going to do

## Checklist

- [x] This pull request has a descriptive title and labels
- [x] This pull request has a minimal description (most was discussed in the issue, but a two-liner description is still desirable)
- [x] Unit tests have been added (if codecov test fails)
- [x] All tests pass



Diff:
diff --git a/.github/workflows/install-from-condalock-file.yml b/.github/workflows/install-from-condalock-file.yml
index 4448895d..96f23fef 100644
--- a/.github/workflows/install-from-condalock-file.yml
+++ b/.github/workflows/install-from-condalock-file.yml
@@ -37,6 +37,13 @@ jobs:
       - run: which python
       - run: python -V
       - run: conda create --name activestorage-fromlock --file conda-linux-64.lock
+      - name: Install development version of NCAS-CMS/Pyfive:wacasoft
+        run: |
+          cd ..
+          git clone https://github.com/NCAS-CMS/pyfive.git
+          cd pyfive
+          git checkout wacasoft
+          pip install -e .
       - run: which python
       - run: pip --version
       - run: pip install -e .
diff --git a/.github/workflows/run-s3-test-push.yml b/.github/workflows/run-s3-test-push.yml
new file mode 100644
index 00000000..bd907375
--- /dev/null
+++ b/.github/workflows/run-s3-test-push.yml
@@ -0,0 +1,85 @@
+# adapted GA workflow from https://github.com/stackhpc/reductionist-rs
+---
+name: S3/Minio Test Latest Python
+
+on:
+  push:
+
+# Required shell entrypoint to have properly configured bash shell
+defaults:
+  run:
+    shell: bash -l {0}
+
+jobs:
+  linux-test:
+    runs-on: "ubuntu-latest"
+    strategy:
+      matrix:
+        python-version: ["3.13"]
+      fail-fast: false
+    name: Linux Python ${{ matrix.python-version }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - uses: conda-incubator/setup-miniconda@v3
+        with:
+          python-version: ${{ matrix.python-version }}
+          miniforge-version: "latest"
+          use-mamba: true
+          mamba-version: "2.0.5"  # https://github.com/conda-incubator/setup-miniconda/issues/392
+      - name: Get conda and Python versions
+        run: |
+          conda --version
+          python -V
+      - name: Export proxy
+        run: |
+          echo 'USE_S3 = True' >> activestorage/config.py
+      - name: Start minio object storage
+        run: tests/s3_exploratory/minio_scripts/minio-start
+      - name: Wait for minio object storage to start
+        run: |
+          until curl -if http://localhost:9001; do
+            sleep 1;
+          done
+      - name: Run Reductionist container
+        run: docker run -it --detach --rm --net=host --name reductionist ghcr.io/stackhpc/reductionist-rs:latest
+      - uses: conda-incubator/setup-miniconda@v3
+        with:
+          activate-environment: activestorage-minio
+          environment-file: environment.yml
+          python-version: ${{ matrix.python-version }}
+          miniforge-version: "latest"
+          use-mamba: true
+          mamba-version: "2.0.5"  # https://github.com/conda-incubator/setup-miniconda/issues/392
+      - name: Install development version of NCAS-CMS/Pyfive:wacasoft
+        run: |
+          cd ..
+          git clone https://github.com/NCAS-CMS/pyfive.git
+          cd pyfive
+          git checkout wacasoft
+          pip install -e .
+      - name: Install PyActiveStorage
+        run: |
+          conda --version
+          python -V
+          which python
+          pip install -e .
+      - name: Run tests
+        run: |
+          pytest -n 2
+      - name: Run S3 exploratory tests
+        run: |
+          pytest tests/s3_exploratory/test_s3_reduction.py --html=test-reports/s3-exploratory-report.html
+        if: always()
+      - name: Install pytest-monitor
+        run: pip install pytest-monitor
+      - name: Run S3 performance tests
+        run: |
+          pytest tests/s3_exploratory/test_s3_arrange_files.py
+          pytest tests/s3_exploratory/test_s3_performance.py --db ../.pymon
+      - name: Analyze S3 and local test performance
+        run: python tests/s3_exploratory/parse_pymon.py
+      - name: Stop minio object storage
+        run: tests/s3_exploratory/minio_scripts/minio-stop
+        if: always()
diff --git a/.github/workflows/run-tests.yml b/.github/workflows/run-tests.yml
index bb5719c9..7f3926ef 100644
--- a/.github/workflows/run-tests.yml
+++ b/.github/workflows/run-tests.yml
@@ -4,7 +4,6 @@ on:
   push:
     branches:
       - main
-      - pyfive
   schedule:
     - cron: '0 0 * * *'  # nightly
 
@@ -78,4 +77,4 @@ jobs:
       - run: conda list
       - run: mamba install -c conda-forge git
       - run: pip install -e .
-      - run: pytest
+      - run: pytest -n 2
diff --git a/.github/workflows/test_s3_minio.yml b/.github/workflows/test_s3_minio.yml
index 4c9a3ef5..0b3553fa 100644
--- a/.github/workflows/test_s3_minio.yml
+++ b/.github/workflows/test_s3_minio.yml
@@ -5,9 +5,7 @@ name: S3/Minio Test
 on:
   push:
     branches:
-      - main  # keep this at all times
-      - pyfive
-  pull_request:
+      - main
   schedule:
     - cron: '0 0 * * *'  # nightly
 
@@ -73,7 +71,7 @@ jobs:
           pip install -e .
       - name: Run tests
         run: |
-          pytest
+          pytest -n 2
       - name: Run S3 exploratory tests
         run: |
           pytest tests/s3_exploratory/test_s3_reduction.py --html=test-reports/s3-exploratory-report.html
diff --git a/activestorage/active.py b/activestorage/active.py
index e9eed0e9..e878ede0 100644
--- a/activestorage/active.py
+++ b/activestorage/active.py
@@ -1,9 +1,11 @@
 import concurrent.futures
 import os
+import fsspec
 import numpy as np
 import pathlib
 import urllib
 import pyfive
+import requests
 import s3fs
 import time
 
@@ -17,6 +19,33 @@
 from activestorage.hdf2numcodec import decode_filters
 
 
+def return_storage_type(uri):
+    """
+    Extract the gateway-protocol to infer what type of storage
+    """
+    try:
+        resp = requests.head(uri)
+    except requests.exceptions.MissingSchema:  # eg local file
+        return
+    except requests.exceptions.InvalidSchema:  # eg Minio file s3://pyactivestorage/common_cl_a.nc
+        if not uri.startswith("s3:"):
+            return
+        else:
+            return "s3"
+    except requests.exceptions.ConnectionError as exc:  # eg invalid link or offline
+        print(exc)
+        return
+    response = resp.headers
+
+    # https files on NGINX don't have "gateway-protocol" key
+    if "gateway-protocol" in response:
+        if response["gateway-protocol"] == "s3":
+            print("Gateway protocol indicates S3 storage.")
+            return "s3"
+    else:
+        return "https"
+
+
 def load_from_s3(uri, storage_options=None):
     """
     Load a netCDF4-like object from S3.
@@ -50,6 +79,19 @@ def load_from_s3(uri, storage_options=None):
     return ds
 
 
+def load_from_https(uri):
+    """
+    Load a pyfive.high_level.Dataset from a
+    netCDF4 file on an https server (NGINX).
+    """
+    #TODO need to test if NGINX server behind https://
+    fs = fsspec.filesystem('http')
+    http_file = fs.open(uri, 'rb')
+    ds = pyfive.File(http_file)
+    print(f"Dataset loaded from https with Pyfive: {uri}")
+    return ds
+
+
 def get_missing_attributes(ds):
     """" 
     Load all the missing attributes we need from a netcdf file
@@ -140,6 +182,24 @@ def __init__(
             self.ds = dataset
         self.uri = dataset
 
+        # determine the storage_type
+        # based on what we have available
+        if not storage_type:
+            if not input_variable:
+                check_uri = self.uri
+            else:
+                check_uri = self.ds.id._filename
+
+            # "special" case when we have to deal
+            # with storage_options['client_kwargs']["endpoint_url"]
+            if storage_options is not None and 'client_kwargs' in storage_options:
+                if "endpoint_url" in storage_options['client_kwargs']:
+                    base_url = storage_options['client_kwargs']["endpoint_url"]
+                    if not input_variable:
+                        check_uri = os.path.join(base_url, self.uri)
+                    else:
+                        check_uri = os.path.join(base_url, self.ds.id._filename)
+            storage_type = return_storage_type(check_uri)
 
         # still allow for a passable storage_type
         # for special cases eg "special-POSIX" ie DDN
@@ -152,6 +212,8 @@ def __init__(
             self.filename = self.ds
         elif input_variable and self.storage_type == "s3":
             self.filename = self.ds.id._filename
+        elif input_variable and self.storage_type == "https":
+            self.filename = self.ds
 
         # get storage_options
         self.storage_options = storage_options
@@ -198,6 +260,8 @@ def __load_nc_file(self):
             nc = pyfive.File(self.uri)
         elif self.storage_type == "s3":
             nc = load_from_s3(self.uri, self.storage_options)
+        elif self.storage_type == "https":
+            nc = load_from_https(self.uri)
         self.filename = self.uri
         self.ds = nc[ncvar]
 
@@ -518,7 +582,7 @@ def _process_chunk(self, session, ds, chunks, chunk_coords, chunk_selection,
                             method=self.method
             )
 
-        elif self.storage_type == "s3" and self._version==2:
+        elif self.storage_type == "s3" and self._version == 2:
             # S3: pass in pre-configured storage options (credentials)
             # print("S3 rfile is:", self.filename)
             parsed_url = urllib.parse.urlparse(self.filename)
@@ -567,6 +631,31 @@ def _process_chunk(self, session, ds, chunks, chunk_coords, chunk_selection,
                                                        chunk_selection,
                                                        axis,
                                                        operation=self._method)
+        # this is for testing ONLY until Reductionist is able to handle https
+        # located files; after that, we can pipe any regular https file through
+        # to Reductionist, provided the https server is "closer" to Reductionist
+        elif self.storage_type == "https" and self._version == 2:
+            # build a simple session
+            session = requests.Session()
+            session.auth = (None, None)
+            session.verify = False
+            bucket = "https"  # really doesn't matter
+
+            # note the extra "storage_type" kwarg
+            # this currently makes Reductionist throw a wobbly
+            # E           activestorage.reductionist.ReductionistError: Reductionist error: HTTP 400: {"error": {"message": "request data is not valid", "caused_by": ["Failed to deserialize the JSON body into the target type", "storage_type: unknown field `storage_type`, expected one of `source`, `bucket`, `object`, `dtype`, `byte_order`, `offset`, `size`, `shape`, `order`, `selection`, `compression`, `filters`, `missing` at line 1 column 550"]}}
+            tmp, count = reductionist.reduce_chunk(session,
+                                                   "https://192.171.169.113:8080",
+                                                   self.filename,
+                                                   bucket, self.filename, offset,
+                                                   size, compressor, filters,
+                                                   self.missing, np.dtype(ds.dtype),
+                                                   chunks,
+                                                   ds._order,
+                                                   chunk_selection,
+                                                   axis,
+                                                   operation=self._method,
+                                                   storage_type="https")
         elif self.storage_type=='ActivePosix' and self.version==2:
             # This is where the DDN Fuse and Infinia wrappers go
             raise NotImplementedError
diff --git a/activestorage/reductionist.py b/activestorage/reductionist.py
index 9b9560da..162f4c36 100644
--- a/activestorage/reductionist.py
+++ b/activestorage/reductionist.py
@@ -29,7 +29,7 @@ def get_session(username: str, password: str, cacert: typing.Optional[str]) -> r
 
 def reduce_chunk(session, server, source, bucket, object,
                  offset, size, compression, filters, missing, dtype, shape,
-                 order, chunk_selection, axis, operation):
+                 order, chunk_selection, axis, operation, storage_type=None):
     """Perform a reduction on a chunk using Reductionist.
 
     :param server: Reductionist server URL
@@ -53,11 +53,14 @@ def reduce_chunk(session, server, source, bucket, object,
                             obtained or operated upon.
     :param axis: tuple of the axes to be reduced (non-negative integers)
     :param operation: name of operation to perform
+    :param storage_type: optional testing flag to allow HTTPS reduction
     :returns: the reduced data as a numpy array or scalar
     :raises ReductionistError: if the request to Reductionist fails
     """
 
-    request_data = build_request_data(source, bucket, object, offset, size, compression, filters, missing, dtype, shape, order, chunk_selection, axis)
+    request_data = build_request_data(source, bucket, object, offset, size, compression,
+                                      filters, missing, dtype, shape, order, chunk_selection,
+                                      axis, storage_type=storage_type)
     if DEBUG:
         print(f"Reductionist request data dictionary: {request_data}")
     api_operation = "sum" if operation == "mean" else operation or "select"
@@ -137,7 +140,7 @@ def encode_missing(missing):
 
 def build_request_data(source: str, bucket: str, object: str, offset: int,
                        size: int, compression, filters, missing, dtype, shape,
-                       order, selection, axis) -> dict:
+                       order, selection, axis, storage_type=None) -> dict:
     """Build request data for Reductionist API."""
     request_data = {
         'source': source,
@@ -148,6 +151,7 @@ def build_request_data(source: str, bucket: str, object: str, offset: int,
         'offset': int(offset),
         'size': int(size),
         'order': order,
+        'storage_type': storage_type,
     }
     if shape:
         request_data["shape"] = shape
diff --git a/activestorage/storage.py b/activestorage/storage.py
index 27a1a794..7f555efb 100644
--- a/activestorage/storage.py
+++ b/activestorage/storage.py
@@ -1,4 +1,5 @@
 """Active storage module."""
+import fsspec
 import numpy as np
 import pyfive
 
@@ -35,18 +36,33 @@ def reduce_chunk(rfile,
         #FIXME: for the moment, open the file every time ... we might want to do that, or not
         # we could just use an instance of pyfive.high_level.Dataset.id
         # passed directly from active.py, as below
-        with open(rfile,'rb') as open_file:
-            # get the data
-            chunk = read_block(open_file, offset, size)
-            # reverse any compression and filters
-            chunk = filter_pipeline(chunk, compression, filters)
-            # make it a numpy array of bytes
-            chunk = ensure_ndarray(chunk)
-            # convert to the appropriate data type
-            chunk = chunk.view(dtype)
-            # sort out ordering and convert to the parent hyperslab dimensions
-            chunk = chunk.reshape(-1, order='A')
-            chunk = chunk.reshape(shape, order=order)
+        try:
+            with open(rfile,'rb') as open_file:
+                # get the data
+                chunk = read_block(open_file, offset, size)
+                # reverse any compression and filters
+                chunk = filter_pipeline(chunk, compression, filters)
+                # make it a numpy array of bytes
+                chunk = ensure_ndarray(chunk)
+                # convert to the appropriate data type
+                chunk = chunk.view(dtype)
+                # sort out ordering and convert to the parent hyperslab dimensions
+                chunk = chunk.reshape(-1, order='A')
+                chunk = chunk.reshape(shape, order=order)
+        except FileNotFoundError:  # could a https file
+            fs = fsspec.filesystem('http')
+            with fs.open(rfile, 'rb') as open_file:
+                # get the data
+                chunk = read_block(open_file, offset, size)
+                # reverse any compression and filters
+                chunk = filter_pipeline(chunk, compression, filters)
+                # make it a numpy array of bytes
+                chunk = ensure_ndarray(chunk)
+                # convert to the appropriate data type
+                chunk = chunk.view(dtype)
+                # sort out ordering and convert to the parent hyperslab dimensions
+                chunk = chunk.reshape(-1, order='A')
+                chunk = chunk.reshape(shape, order=order)
     else:
             class storeinfo: pass
             storeinfo.byte_offset = offset
diff --git a/tests/test_bigger_data.py b/tests/test_bigger_data.py
index f423218d..aa78f907 100644
--- a/tests/test_bigger_data.py
+++ b/tests/test_bigger_data.py
@@ -146,6 +146,31 @@ def test_ps(tmp_path):
     np.testing.assert_array_equal(mean_result, result2["sum"]/result2["n"])
 
 
+def test_ps_implicit_storage_type(tmp_path):
+    """
+    Test a Minio S3 file that's not behind a HTTPS URI
+    s3://pyactivestorage/common_cl_a.nc
+    """
+    ncfile = save_cl_file_with_a(tmp_path)
+    active = Active(ncfile, "ps")
+    active._version = 0
+    d = active[4:5, 1:2]
+    mean_result = np.mean(d)
+
+    active = Active(ncfile, "ps")
+    active._version = 2
+    active.method = "mean"
+    active.components = True
+    result2 = active[4:5, 1:2]
+    print(result2, ncfile)
+    # expect {'sum': array([[[22.]]]), 'n': array([[[4]]])}
+    # check for typing and structure
+    np.testing.assert_array_equal(result2["sum"], np.array([[[22.]]]))
+    np.testing.assert_array_equal(result2["n"], np.array([[[4]]]))
+    # check for active
+    np.testing.assert_array_equal(mean_result, result2["sum"]/result2["n"])
+
+
 def test_native_emac_model_fails(test_data_path):
     """
     An example of netCDF file that doesn't work
diff --git a/tests/test_real_https.py b/tests/test_real_https.py
new file mode 100644
index 00000000..1b658596
--- /dev/null
+++ b/tests/test_real_https.py
@@ -0,0 +1,116 @@
+import os
+import numpy as np
+import activestorage
+import pytest
+
+from activestorage.active import Active
+from activestorage.active import load_from_https
+
+
+@pytest.mark.skip(reason="save time: test_https_implicit_storage is more general.")
+def test_https():
+    """Run a true test with a https FILE."""
+    test_file_uri = "https://esgf.ceda.ac.uk/thredds/fileServer/esg_cmip6/CMIP6/AerChemMIP/MOHC/UKESM1-0-LL/ssp370SST-lowNTCF/r1i1p1f2/Amon/cl/gn/latest/cl_Amon_UKESM1-0-LL_ssp370SST-lowNTCF_r1i1p1f2_gn_205001-209912.nc"
+
+    active = Active(test_file_uri, "cl", storage_type="https")
+    active._version = 1
+    active._method = "min"
+    result = active[0:3, 4:6, 7:9]
+    print("Result is", result)
+    assert result == np.array([0.6909787], dtype="float32")
+
+
+@pytest.mark.skip(reason="save time: 2xdata = 2xtime compared to test_https.")
+def test_https_100years():
+    """Run a true test with a https FILE."""
+    test_file_uri = "https://esgf.ceda.ac.uk/thredds/fileServer/esg_cmip6/CMIP6/CMIP/MOHC/UKESM1-1-LL/historical/r1i1p1f2/Amon/pr/gn/latest/pr_Amon_UKESM1-1-LL_historical_r1i1p1f2_gn_195001-201412.nc"
+    active = Active(test_file_uri, "pr")
+    active._version = 1
+    active._method = "min"
+    result = active[0:3, 4:6, 7:9]
+    print("Result is", result)
+    assert result == np.array([5.4734613e-07], dtype="float32")
+
+
+def test_https_reductionist():
+    """Run a true test with a https FILE."""
+    test_file_uri = "https://esgf.ceda.ac.uk/thredds/fileServer/esg_cmip6/CMIP6/AerChemMIP/MOHC/UKESM1-0-LL/ssp370SST-lowNTCF/r1i1p1f2/Amon/cl/gn/latest/cl_Amon_UKESM1-0-LL_ssp370SST-lowNTCF_r1i1p1f2_gn_205001-209912.nc"
+
+    # added storage_type in request_data dict; Reductionist not liking it
+    # E           activestorage.reductionist.ReductionistError: Reductionist error: HTTP 400: {"error": {"message": "request data is not valid", "caused_by": ["Failed to deserialize the JSON body into the target type", "storage_type: unknown field `storage_type`, expected one of `source`, `bucket`, `object`, `dtype`, `byte_order`, `offset`, `size`, `shape`, `order`, `selection`, `compression`, `filters`, `missing` at line 1 column 550"]}}
+    with pytest.raises(activestorage.reductionist.ReductionistError):
+        active = Active(test_file_uri, "cl")
+        active._version = 2
+        active._method = "min"
+        result = active[0:3, 4:6, 7:9]
+        print("Result is", result)
+        assert result == np.array([0.6909787], dtype="float32")
+
+
+def test_https_implicit_storage():
+    """Run a true test with a https FILE."""
+    test_file_uri = "https://esgf.ceda.ac.uk/thredds/fileServer/esg_cmip6/CMIP6/AerChemMIP/MOHC/UKESM1-0-LL/ssp370SST-lowNTCF/r1i1p1f2/Amon/cl/gn/latest/cl_Amon_UKESM1-0-LL_ssp370SST-lowNTCF_r1i1p1f2_gn_205001-209912.nc"
+
+    active = Active(test_file_uri, "cl")
+    active._version = 1
+    active._method = "min"
+    result = active[0:3, 4:6, 7:9]
+    print("Result is", result)
+    assert result == np.array([0.6909787], dtype="float32")
+
+
+def test_https_implicit_storage_file_not_found():
+    """
+    Run a true test with a https FILE that is not found.
+    Code raises a very descriptive exception via fsspec.
+    Keep test to capture any changes in behaviour.
+    """
+    test_file_uri = "https://esgf.ceda.ac.uk/thredds/fileServer/esg_cmip6/CMIP6/AerChemMIP/MOHC/UKESM1-0-LL/ssp370SST-lowNTCF/r1i1p1f2/Amon/cl/gn/latest/cl_Amon_UKESM1-0-LL_ssp370SST-lowNTCF_r1i1p1f2_gn_205001-209912.ncx"
+
+    with pytest.raises(FileNotFoundError):
+        active = Active(test_file_uri, "cl")
+        active._version = 1
+        active._method = "min"
+        result = active[0:3, 4:6, 7:9]
+
+
+def test_https_implicit_storage_wrong_url():
+    """
+    Run a true test with a bogus URL.
+    """
+    test_file_uri = "https://esgf.cedacow.ac.uk/thredds/fileServer/esg_cmip6/CMIP6/AerChemMIP/MOHC/UKESM1-0-LL/ssp370SST-lowNTCF/r1i1p1f2/Amon/cl/gn/latest/cl_Amon_UKESM1-0-LL_ssp370SST-lowNTCF_r1i1p1f2_gn_205001-209912.nc"
+
+    with pytest.raises(ValueError):
+        active = Active(test_file_uri, "cl")
+        active._version = 1
+        active._method = "min"
+        result = active[0:3, 4:6, 7:9]
+
+
+@pytest.mark.skip(reason="save time: test_https_dataset_implicit_storage is more general.")
+def test_https_dataset():
+    """Run a true test with a https DATASET."""
+    test_file_uri = "https://esgf.ceda.ac.uk/thredds/fileServer/esg_cmip6/CMIP6/AerChemMIP/MOHC/UKESM1-0-LL/ssp370SST-lowNTCF/r1i1p1f2/Amon/cl/gn/latest/cl_Amon_UKESM1-0-LL_ssp370SST-lowNTCF_r1i1p1f2_gn_205001-209912.nc"
+    dataset = load_from_https(test_file_uri)
+    av = dataset['cl']
+
+    active = Active(av, storage_type="https")
+    active._version = 1
+    active._method = "min"
+    result = active[0:3, 4:6, 7:9]
+    print("Result is", result)
+    assert result == np.array([0.6909787], dtype="float32")
+
+
+def test_https_dataset_implicit_storage():
+    """Run a true test with a https DATASET."""
+    test_file_uri = "https://esgf.ceda.ac.uk/thredds/fileServer/esg_cmip6/CMIP6/AerChemMIP/MOHC/UKESM1-0-LL/ssp370SST-lowNTCF/r1i1p1f2/Amon/cl/gn/latest/cl_Amon_UKESM1-0-LL_ssp370SST-lowNTCF_r1i1p1f2_gn_205001-209912.nc"
+    dataset = load_from_https(test_file_uri)
+    av = dataset['cl']
+
+    active = Active(av)
+    active._version = 1
+    active._method = "min"
+    result = active[0:3, 4:6, 7:9]
+    print("Result is", result)
+    assert result == np.array([0.6909787], dtype="float32")
diff --git a/tests/test_real_s3.py b/tests/test_real_s3.py
index 2a3f0d50..3164a8eb 100644
--- a/tests/test_real_s3.py
+++ b/tests/test_real_s3.py
@@ -4,16 +4,16 @@
 from activestorage.active import Active
 from activestorage.active import load_from_s3
 
+
 S3_BUCKET = "bnl"
 
-# TODO Remove after full testing and right before deployment
 def test_s3_dataset():
     """Run somewhat as the 'gold' test."""
     storage_options = {
         'key': "f2d55c6dcfc7618b2c34e00b58df3cef",
         'secret': "$/'#M{0{/4rVhp%n^(XeX$q@y#&(NM3W1->~N.Q6VP.5[@bLpi='nt]AfH)>78pT",
-        'client_kwargs': {'endpoint_url': "https://uor-aces-o.s3-ext.jc.rl.ac.uk"},  # old proxy
-        # 'client_kwargs': {'endpoint_url': "https://uor-aces-o.ext.proxy.jc.rl.ac.uk"},  # new proxy
+        # 'client_kwargs': {'endpoint_url': "https://uor-aces-o.s3-ext.jc.rl.ac.uk"},  # old proxy
+        'client_kwargs': {'endpoint_url': "https://uor-aces-o.ext.proxy.jc.rl.ac.uk"},  # new proxy
     }
     active_storage_url = "https://192.171.169.113:8080"
     # bigger_file = "ch330a.pc19790301-bnl.nc"  # 18GB 3400 HDF5 chunks
@@ -25,18 +25,48 @@ def test_s3_dataset():
         bigger_file
     )
     print("S3 Test file path:", test_file_uri)
+
+    # file: explicit storage_type
+    active = Active(test_file_uri, 'UM_m01s16i202_vn1106',
+                    storage_type="s3",
+                    storage_options=storage_options,
+                    active_storage_url=active_storage_url)
+    active._version = 2
+    active._method = "min"
+    result = active[0:3, 4:6, 7:9]  # standardized slice
+    print("Result is", result)
+    assert result == 5098.625
+
+    # file: implicit storage_type
+    active = Active(test_file_uri, 'UM_m01s16i202_vn1106',
+                    storage_options=storage_options,
+                    active_storage_url=active_storage_url)
+    active._version = 2
+    active._method = "min"
+    result = active[0:3, 4:6, 7:9]  # standardized slice
+    print("Result is", result)
+    assert result == 5098.625
+
+    # load dataset
     dataset = load_from_s3(test_file_uri, storage_options=storage_options)
     av = dataset['UM_m01s16i202_vn1106']
 
-    # big file bnl: 18GB/3400 HDF5 chunks; def: 17GB/64 HDF5 chunks
+    # dataset: explicit storage_type
     active = Active(av, storage_type="s3",
                     storage_options=storage_options,
                     active_storage_url=active_storage_url)
     active._version = 2
     active._method = "min"
-
-    # result = active[:]
     result = active[0:3, 4:6, 7:9]  # standardized slice
+    print("Result is", result)
+    assert result == 5098.625
 
+    # dataset: implicit storage_type
+    active = Active(av,
+                    storage_options=storage_options,
+                    active_storage_url=active_storage_url)
+    active._version = 2
+    active._method = "min"
+    result = active[0:3, 4:6, 7:9]  # standardized slice
     print("Result is", result)
     assert result == 5098.625
diff --git a/tests/unit/test_active.py b/tests/unit/test_active.py
index 1e6fd805..004c11de 100644
--- a/tests/unit/test_active.py
+++ b/tests/unit/test_active.py
@@ -161,6 +161,7 @@ def test_get_endpoint_url():
     }
     uri = "tests/test_data/cesm2_native.nc"
     ncvar = "TREFHT"
-    active = Active(uri, ncvar=ncvar, storage_options=storage_options)
+    active = Active(uri, ncvar=ncvar, storage_type="s3",
+                    storage_options=storage_options)
     ep_url = Active._get_endpoint_url(active)
     assert ep_url == "https://cow.moo"


Reviews:
user: login: valeriupredoi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-11T17:21:38Z
user: login: valeriupredoi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-11T17:23:16Z
user: login: valeriupredoi
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-12T09:47:50Z
user: login: bnlawrence
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-12T10:06:10Z
user: login: davidhassell
body: Hi V, looks good to me on the basis of running the tests locally (all pass), and skim through the code changes new tests - saw nothing of worry!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-20T12:07:25Z

Review comments:
diff_hunk: @@ -508,6 +567,26 @@ def _process_chunk(self, session, ds, chunks, chunk_coords, chunk_selection, cou
                                                        ds._order,
                                                        chunk_selection,
                                                        operation=self._method)
+        # this is for testing ONLY until Reductionist is able to handle https
+        # located files; after that, we can pipe any regular https file through
+        # to Reductionist, provided the https server is "closer" to Reductionist
+        elif self.storage_type == "https-Reductionist" and self._version==2:
+            # build a simple session
+            session = requests.Session()
+            session.auth = (None, None)
+            session.verify = False
+            bucket = "https"
+
+            tmp, count = reductionist.reduce_chunk(session,
+                                                   "https://192.171.169.113:8080",
+                                                   self.filename,
+                                                   bucket, self.filename, offset,
+                                                   size, compressor, filters,
+                                                   self.missing, np.dtype(ds.dtype),
+                                                   chunks,
+                                                   ds._order,
+                                                   chunk_selection,
+                                                   operation=self._method)
path: activestorage/active.py
user: login: valeriupredoi
body: @sd109 here is a used test case that should work if there is backend implementation in Reductionist: this really changes nothing in the Reductionist API, but can do the switch to the backend Reductionist loading an https file instead of an S3 file simply by calling `bucket="https"`, the rest is all the same, bar `object` is a duplicate of `source`, both being the full ULR to the file, here https://esgf.ceda.ac.uk/thredds/fileServer/esg_cmip6/CMIP6/AerChemMIP/MOHC/UKESM1-0-LL/ssp370SST-lowNTCF/r1i1p1f2/Amon/cl/gn/latest/cl_Amon_UKESM1-0-LL_ssp370SST-lowNTCF_r1i1p1f2_gn_205001-209912.nc - what do you think?
created_at: 2025-03-11T17:21:02Z
author_association: COLLABORATOR
diff_hunk: @@ -508,6 +567,26 @@ def _process_chunk(self, session, ds, chunks, chunk_coords, chunk_selection, cou
                                                        ds._order,
                                                        chunk_selection,
                                                        operation=self._method)
+        # this is for testing ONLY until Reductionist is able to handle https
+        # located files; after that, we can pipe any regular https file through
+        # to Reductionist, provided the https server is "closer" to Reductionist
+        elif self.storage_type == "https-Reductionist" and self._version==2:
+            # build a simple session
+            session = requests.Session()
+            session.auth = (None, None)
+            session.verify = False
+            bucket = "https"
+
+            tmp, count = reductionist.reduce_chunk(session,
+                                                   "https://192.171.169.113:8080",
+                                                   self.filename,
+                                                   bucket, self.filename, offset,
+                                                   size, compressor, filters,
+                                                   self.missing, np.dtype(ds.dtype),
+                                                   chunks,
+                                                   ds._order,
+                                                   chunk_selection,
+                                                   operation=self._method)
path: activestorage/active.py
user: login: valeriupredoi
body: this currently fails with `activestorage.reductionist.ReductionistError: Reductionist error: HTTP 500: {"error": {"message": "error retrieving object from S3 storage", "caused_by": ["service error", "unhandled error", "Error"]}}` as it should be :grin: 
created_at: 2025-03-11T17:23:16Z
author_association: COLLABORATOR
diff_hunk: @@ -17,6 +19,25 @@
 from activestorage.hdf2numcodec import decode_filters
 
 
+def return_storage_type(uri):
+    """
+    Extract the gateway-protocol to infer what type of storage
+    """
+    try:
+        resp = requests.head(uri)
+    except requests.exceptions.MissingSchema:  # eg local file
+        return
+    response = resp.headers
+
+    # https files on NGINX don't have "gateway-protocol" key
+    if "gateway-protocol" in response:
+        if response["gateway-protocol"] == "s3":
+            print("Gateway protocol indicates S3 storage.")
+            return "s3"
+    else:
+        return "https"
path: activestorage/active.py
user: login: valeriupredoi
body: @bnlawrence this is the bit I was on about on the call
created_at: 2025-03-12T09:47:40Z
author_association: COLLABORATOR
diff_hunk: @@ -17,6 +19,25 @@
 from activestorage.hdf2numcodec import decode_filters
 
 
+def return_storage_type(uri):
+    """
+    Extract the gateway-protocol to infer what type of storage
+    """
+    try:
+        resp = requests.head(uri)
+    except requests.exceptions.MissingSchema:  # eg local file
+        return
+    response = resp.headers
+
+    # https files on NGINX don't have "gateway-protocol" key
+    if "gateway-protocol" in response:
+        if response["gateway-protocol"] == "s3":
+            print("Gateway protocol indicates S3 storage.")
+            return "s3"
+    else:
+        return "https"
path: activestorage/active.py
user: login: bnlawrence
body: That's great. I think we'd need to put the response to that in the json going to reductionist, rather than in the URI itself.
created_at: 2025-03-12T10:06:10Z
author_association: COLLABORATOR

PR comments:
user: login: valeriupredoi
created_at: 2025-03-14T12:09:54Z
body: @bnlawrence @davidhassell this one's ready for your review, please, gents - the https chunk reduction is actually done using threads, since there is now little difference (actually none at all) between `_version = 1` and `_version = 2` (I will harmoize those options ie cleanup in a near future PR) :beer: 
author_association: COLLABORATOR
user: login: codecov-commenter
created_at: 2025-03-14T15:42:51Z
body: ## [Codecov](https://app.codecov.io/gh/NCAS-CMS/PyActiveStorage/pull/245?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAS-CMS) Report
Attention: Patch coverage is `95.52239%` with `3 lines` in your changes missing coverage. Please review.
> Project coverage is 91.18%. Comparing base [(`0e13492`)](https://app.codecov.io/gh/NCAS-CMS/PyActiveStorage/commit/0e13492b1de37f4a1af36c7e85981fe75e185484?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAS-CMS) to head [(`7891742`)](https://app.codecov.io/gh/NCAS-CMS/PyActiveStorage/commit/7891742206e1a87f50aa66f940942201ca19981b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAS-CMS).

| [Files with missing lines](https://app.codecov.io/gh/NCAS-CMS/PyActiveStorage/pull/245?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAS-CMS) | Patch % | Lines |
|---|---|---|
| [activestorage/active.py](https://app.codecov.io/gh/NCAS-CMS/PyActiveStorage/pull/245?src=pr&el=tree&filepath=activestorage%2Factive.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAS-CMS#diff-YWN0aXZlc3RvcmFnZS9hY3RpdmUucHk=) | 93.75% | [3 Missing :warning: ](https://app.codecov.io/gh/NCAS-CMS/PyActiveStorage/pull/245?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAS-CMS) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #245      +/-   ##
==========================================
+ Coverage   90.98%   91.18%   +0.19%     
==========================================
  Files           7        7              
  Lines         577      635      +58     
==========================================
+ Hits          525      579      +54     
- Misses         52       56       +4     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/NCAS-CMS/PyActiveStorage/pull/245?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAS-CMS).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NCAS-CMS).

<details><summary>🚀 New features to boost your workflow: </summary>

- ❄ [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: valeriupredoi
created_at: 2025-03-20T17:23:45Z
body: very many thanks @davidhassell :beer: 
author_association: COLLABORATOR