Repository: kedro-org/kedro

Issue:
Catch error for issues with reading/writing data to object stores (e.g. S3)
The original Discord discussion https://discord.com/channels/778216384475693066/989568455512555540

# Context
> williamc — 06/23/2022
Let's say I just cloned my kedro project repo to another machine, and its datasets are versioned and configured to use S3 for storage. If I try to run a pipeline that depends on those datasets I get the infamous kedro.io.core.VersionNotFoundError . Bucket has versions all the way up to 2022-06-07T22.04.39.460Z/ and the error says 2022-06-23T16.20.52.945Z. Is this the intended behavior? Thanks

# Solution
User report that this is resolved after he changes the permission to `s3:*` from `s3:*Object`. I suspect we need  `ListObjects` or `ListObjectsV2` only.

# What's the problem?
I expect it to throw an error like `Insufficient permissions to list objects`, these logic is likely handled by `fsspec` and the `glob_function` we pass into `VersionedDataSet`. Some more investigation is needed

# Action:
1. Show a better error like `Insufficient permissions to list objects` instead of `VersionedNotFound`? This is confusing because the user can see the files but `kedro` would complain the file isn't there. It's not clear currently if the error doesn't come from Kedro directly but from e.g. fsspec
2. Check what kind of Policy are needed for `kedro` to work? (p.s. `*` is definitely not a good suggestion!) and document this clearly







Discussion:

Resolving patch:
diff --git a/kedro/io/core.py b/kedro/io/core.py
--- a/kedro/io/core.py
+++ b/kedro/io/core.py
@@ -538,9 +538,16 @@ def _fetch_latest_load_version(self) -> str:
         most_recent = next(
             (path for path in version_paths if self._exists_function(path)), None
         )
-
+        protocol = getattr(self, "_protocol", None)
         if not most_recent:
-            raise VersionNotFoundError(f"Did not find any versions for {self}")
+            if protocol in CLOUD_PROTOCOLS:
+                message = (
+                    f"Did not find any versions for {self}. This could be "
+                    f"due to insufficient permission."
+                )
+            else:
+                message = f"Did not find any versions for {self}"
+            raise VersionNotFoundError(message)
 
         return PurePath(most_recent).parent.name
 


Test patch:
diff --git a/tests/io/test_data_catalog.py b/tests/io/test_data_catalog.py
--- a/tests/io/test_data_catalog.py
+++ b/tests/io/test_data_catalog.py
@@ -19,7 +19,7 @@
     LambdaDataSet,
     MemoryDataSet,
 )
-from kedro.io.core import VERSION_FORMAT, generate_timestamp
+from kedro.io.core import VERSION_FORMAT, Version, generate_timestamp
 
 
 @pytest.fixture
@@ -652,3 +652,14 @@ def test_replacing_nonword_characters(self):
         assert "ds2_spark" in catalog.datasets.__dict__
         assert "ds3__csv" in catalog.datasets.__dict__
         assert "jalapeño" in catalog.datasets.__dict__
+
+    def test_no_versions_with_cloud_protocol(self):
+        """Check the error if no versions are available for load from cloud storage"""
+        version = Version(load=None, save=None)
+        versioned_dataset = CSVDataSet("s3://bucket/file.csv", version=version)
+        pattern = re.escape(
+            f"Did not find any versions for {versioned_dataset}. "
+            f"This could be due to insufficient permission."
+        )
+        with pytest.raises(DataSetError, match=pattern):
+            versioned_dataset.load()