Repository: kedro-org/kedro

Issue:
Support full URIs in `abfss://` paths
## Description
I found a breaking change in Azure Data Lake Storage Gen 2 paths (`abfss://` protocol) when upgrading from kedro 0.17.7 to 0.18.4 that was undocumented, I was wondering if that was intentional and by design or if it's a bug that should be fixed. 

In Kedro 0.17.7 I had a `JSONDataSet` (it would apply for any `AbstractVersionedDataSet`) that had the path set in the yml catalog configuration using the [abfss URI syntax](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction-abfs-uri#uri-syntax). It looked like: `abfss://mycontainer@mystorageaccount.dfs.core.windows.net/mypath`, and worked fine

After upgrading to Kedro 0.18.4, it started failing, because it wouldn't find any of the versions of my file. I believe this is cause by the addition of `abfss` to the list of `CLOUD_PROTOCOLS` in fa02d9b6aed350251e2c136fd593978ceca2b9c3:

https://github.com/kedro-org/kedro/blob/adc7e9f83b3938492c7fa1e897b1e8ff56d737d0/kedro/io/core.py#L31

This causes the filepath I introduced to be altered in this part of the `AbstractVersionedDataSet` source code: 

https://github.com/kedro-org/kedro/blob/adc7e9f83b3938492c7fa1e897b1e8ff56d737d0/kedro/io/core.py#L688-L692

From what I see in the following kedro test, `abfss` implementation was designed to just use the path and not the full URI that includes the container and storage account, but I think supporting full URIs would also be great: 

https://github.com/kedro-org/kedro/blob/adc7e9f83b3938492c7fa1e897b1e8ff56d737d0/tests/io/test_core.py#L67-L70

I don't understand well why the `CLOUD_PROTOCOLS` filepaths prostprocessing is there, and how should we fix this without affecting other protocols and not breaking the "simple path" approach like in the test above. 
How should we update the `CLOUD_PROTOCOLS` implementation for full URIs to work without breaking other protocols?

Support full URIs in `abfss://` paths
## Description
I found a breaking change in Azure Data Lake Storage Gen 2 paths (`abfss://` protocol) when upgrading from kedro 0.17.7 to 0.18.4 that was undocumented, I was wondering if that was intentional and by design or if it's a bug that should be fixed. 

In Kedro 0.17.7 I had a `JSONDataSet` (it would apply for any `AbstractVersionedDataSet`) that had the path set in the yml catalog configuration using the [abfss URI syntax](https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction-abfs-uri#uri-syntax). It looked like: `abfss://mycontainer@mystorageaccount.dfs.core.windows.net/mypath`, and worked fine

After upgrading to Kedro 0.18.4, it started failing, because it wouldn't find any of the versions of my file. I believe this is cause by the addition of `abfss` to the list of `CLOUD_PROTOCOLS` in fa02d9b6aed350251e2c136fd593978ceca2b9c3:

https://github.com/kedro-org/kedro/blob/adc7e9f83b3938492c7fa1e897b1e8ff56d737d0/kedro/io/core.py#L31

This causes the filepath I introduced to be altered in this part of the `AbstractVersionedDataSet` source code: 

https://github.com/kedro-org/kedro/blob/adc7e9f83b3938492c7fa1e897b1e8ff56d737d0/kedro/io/core.py#L688-L692

From what I see in the following kedro test, `abfss` implementation was designed to just use the path and not the full URI that includes the container and storage account, but I think supporting full URIs would also be great: 

https://github.com/kedro-org/kedro/blob/adc7e9f83b3938492c7fa1e897b1e8ff56d737d0/tests/io/test_core.py#L67-L70

I don't understand well why the `CLOUD_PROTOCOLS` filepaths prostprocessing is there, and how should we fix this without affecting other protocols and not breaking the "simple path" approach like in the test above. 
How should we update the `CLOUD_PROTOCOLS` implementation for full URIs to work without breaking other protocols?



Discussion:

Resolving patch:
diff --git a/kedro/io/core.py b/kedro/io/core.py
--- a/kedro/io/core.py
+++ b/kedro/io/core.py
@@ -685,11 +685,14 @@ def _parse_filepath(filepath: str) -> Dict[str, str]:
 
     options = {"protocol": protocol, "path": path}
 
-    if parsed_path.netloc:
-        if protocol in CLOUD_PROTOCOLS:
-            host_with_port = parsed_path.netloc.rsplit("@", 1)[-1]
-            host = host_with_port.rsplit(":", 1)[0]
-            options["path"] = host + options["path"]
+    if parsed_path.netloc and protocol in CLOUD_PROTOCOLS:
+        host_with_port = parsed_path.netloc.rsplit("@", 1)[-1]
+        host = host_with_port.rsplit(":", 1)[0]
+        options["path"] = host + options["path"]
+        # Azure Data Lake Storage Gen2 URIs can store the container name in the
+        # 'username' field of a URL (@ syntax), so we need to add it to the path
+        if protocol == "abfss" and parsed_path.username:
+            options["path"] = parsed_path.username + "@" + options["path"]
 
     return options
 


Test patch:
diff --git a/tests/io/test_core.py b/tests/io/test_core.py
--- a/tests/io/test_core.py
+++ b/tests/io/test_core.py
@@ -68,6 +68,13 @@ def test_get_filepath_str(self):
                 "abfss://bucket/file.txt",
                 {"protocol": "abfss", "path": "bucket/file.txt"},
             ),
+            (
+                "abfss://mycontainer@mystorageaccount.dfs.core.windows.net/mypath",
+                {
+                    "protocol": "abfss",
+                    "path": "mycontainer@mystorageaccount.dfs.core.windows.net/mypath",
+                },
+            ),
             (
                 "hdfs://namenode:8020/file.txt",
                 {"protocol": "hdfs", "path": "/file.txt"},