Repository: kubeflow/pipelines

Issue:
ComponentStore fails to include filename when pulling by version or tag
Looking at the comments around `ComponentStore` more closely, I don't think this is a bug, but rather my misunderstanding of the intended behavior: I'll need to adjust how I publish my components. The further question about using ComponentStore to load standard components (in a comment below) still stands. Leaving this here to get clarification:

### What steps did you take:
* stored a Kubeflow component at https://www.example.com/kubeflow/components/gcp/storage/download_dirs/versions/tags/0.1.0/component.yaml

* configured a `ComponentStore`:
```python
components = ComponentStore(
    url_search_prefixes=[ "https://www.example.com/kubeflow/components/" ]
)
```

* fetched a versioned component using:
```python
download_from_gcs_op = components.load_component("gcp/storage/download_dirs", tag="0.1.0")
```

### What happened:
With appropriate debugging, it became apparent this was causing a failure because the ComponentStore is attempting to fetch https://www.example.com/kubeflow/components/gcp/storage/download_dirs/versions/tags/0.1.0 and not https://www.example.com/kubeflow/components/gcp/storage/download_dirs/versions/tags/0.1.0/component.yaml.

### What did you expect to happen:
Component to be retrieved from https://www.example.com/kubeflow/components/gcp/storage/download_dirs/versions/tags/0.1.0/component.yaml.

### Environment:

KFP SDK version: kfp 1.1.0a1


### Anything else you would like to add:
I'm unclear if this is intentional, and tagged components _should_ be stored in files bearing the tag as a filename (and no extension). If not, I could probably do a PR, as the reason for this is pretty clear:

https://github.com/kubeflow/pipelines/blob/master/sdk/python/kfp/components/_component_store.py#L131

Note that the `component_file_name` is in an else block, so is only added if digest and tag are not specified.

/kind bug
<!-- Please include labels by uncommenting them to help us better triage issues, choose from the following -->
<!--
// /area frontend
// /area backend
/area sdk
// /area testing
// /area engprod
-->



Discussion:
As a related point, I'd hoped I could use the default ComponentStore to retrieve tagged versions of standard components, but:

```python
ComponentStore.default_store.load_component(
    "google-cloud/storage/upload_to_explicit_uri", tag="1.0.1"
)
```

attempts to retrieve:

https://raw.githubusercontent.com/kubeflow/pipelines/master/components/google-cloud/storage/upload_to_explicit_uri/versions/tags/1.0.1

rather than the correct path, which would be:

https://raw.githubusercontent.com/kubeflow/pipelines/1.0.1/components/google-cloud/storage/upload_to_explicit_uri/component.yaml
Perhaps if rather than `url_search_prefixes` we could define URI templates, with `tag` and `digest` as variables?
/assign @chensun @numerology @Ark-kun 
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Is there any update on this? As was mentioned, the fix seems doable, but it's not clear to me if this behavior is intended. 
Are tags/digest supposed to be actual file locations or the source ref?

What is the recommended way to load in versioned components? 
Found an [old answer](https://github.com/kubeflow/pipelines/issues/1025#issuecomment-475808132) from @Ark-kun on the intended behavior for versioning which answers my question- the filename is supposed to be the file hash/tag and stored in sub-directory of the component. 

Unless I'm missing something, I feel like that workflow makes more sense in the context of an external storage system like GCS than with git because you're maintaining all historical versions of the component definitions alongside the source code which seems redundant compared to just referencing the commit in the URL. 
I agree: if you're using a hosted service such as GitHub you're often at their mercy when it comes to the exact URI structure. The more I've thought about this, the more I like being able to define a URI template (RFC 6570), using `{tag}` and/or `{digest}`.

I'd propose this change as a PR, but at a quick look I couldn't see an RFC 6570 package as a dependency, and introducing new dependencies into the SDK felt above my paygrade. There are plenty of options, e.g. https://pypi.org/project/uri-template.

Thoughts?

Resolving patch:
diff --git a/sdk/python/kfp/components/_component_store.py b/sdk/python/kfp/components/_component_store.py
--- a/sdk/python/kfp/components/_component_store.py
+++ b/sdk/python/kfp/components/_component_store.py
@@ -10,6 +10,7 @@
 import requests
 import tempfile
 from typing import Callable, Iterable
+from uritemplate import URITemplate
 from . import _components as comp
 from .structures import ComponentReference
 from ._key_value_store import KeyValueStore
@@ -18,13 +19,34 @@
 
 
 class ComponentStore:
+    """Component store.
+    
+    Enables external components to be loaded by name and digest/tag.
+    
+    Attributes:
+
+        local_search_paths: A list of local directories to include in the search.
+        url_seach_prefixes: A list of URL prefixes to include in the search.
+        uri_search_template: A URI template for components, which may include {name}, {digest} and {tag} variables.
+    """
 
     def __init__(self,
                  local_search_paths=None,
                  url_search_prefixes=None,
-                 auth=None):
-        """Instantiates a ComponentStore."""
+                 auth=None,
+                 uri_search_template=None):
+        """Instantiates a ComponentStore including the specified locations.
+        
+        Args:
+
+            local_search_paths: A list of local directories to include in the search.
+            url_seach_prefixes: A list of URL prefixes to include in the search.
+            auth: Auth object for the requests library. See https://requests.readthedocs.io/en/master/user/authentication/
+            uri_search_template: A URI template for components, which may include {name}, {digest} and {tag} variables.
+        """
         self.local_search_paths = local_search_paths or ['.']
+        if uri_search_template:
+            self.uri_search_template = URITemplate(uri_search_template)
         self.url_search_prefixes = url_search_prefixes or []
         self._auth = auth
 
@@ -154,28 +176,55 @@ def _load_component_spec_in_component_ref(
                     str(component_path))
                 return component_ref
 
+        #Trying URI template
+        if self.uri_search_template:
+            url = self.uri_search_template.expand(
+                name=name, digest=digest, tag=tag)
+            tried_locations.append(url)
+            if self._load_component_spec_from_url(component_ref, url):
+                return component_ref
+
         #Trying URL prefixes
         for url_search_prefix in self.url_search_prefixes:
             url = url_search_prefix + path_suffix
             tried_locations.append(url)
-            try:
-                response = requests.get(
-                    url, auth=self._auth
-                )  #Does not throw exceptions on bad status, but throws on dead domains and malformed URLs. Should we log those cases?
-                response.raise_for_status()
-            except:
-                continue
-            if response.content:
-                # TODO: Verify that the content matches the digest (if specified).
-                component_ref.url = url
-                component_ref.spec = comp._load_component_spec_from_yaml_or_zip_bytes(
-                    response.content)
+            if self._load_component_spec_from_url(component_ref, url):
                 return component_ref
 
         raise RuntimeError(
             'Component {} was not found. Tried the following locations:\n{}'
             .format(name, '\n'.join(tried_locations)))
 
+    def _load_component_spec_from_url(self, component_ref, url) -> bool:
+        """Loads component spec from a URL.
+
+        On success, the url and spec attributes of the component_ref arg will be populated.
+
+        Args:
+            component_ref: the component whose spec to load.
+            url: the location from which to obtain the component spec.
+
+        Returns:
+            True if the component was retrieved and non-empty; otherwise False.
+        """
+
+        try:
+            response = requests.get(
+                url, auth=self._auth
+            )  #Does not throw exceptions on bad status, but throws on dead domains and malformed URLs. Should we log those cases?
+            response.raise_for_status()
+        except:
+            return False
+
+        if response.content:
+            # TODO: Verify that the content matches the digest (if specified).
+            component_ref.url = url
+            component_ref.spec = comp._load_component_spec_from_yaml_or_zip_bytes(
+                response.content)
+            return True
+
+        return False
+
     def _load_component_from_ref(self,
                                  component_ref: ComponentReference) -> Callable:
         component_ref = self._load_component_spec_in_component_ref(
@@ -339,6 +388,7 @@ def _list_candidate_component_uris_from_github_repo(url_search_prefix: str,
     local_search_paths=[
         '.',
     ],
+    uri_search_template='https://raw.githubusercontent.com/kubeflow/pipelines/{tag}/components/{name}/component.yaml',
     url_search_prefixes=[
         'https://raw.githubusercontent.com/kubeflow/pipelines/master/components/'
     ],
diff --git a/sdk/python/setup.py b/sdk/python/setup.py
--- a/sdk/python/setup.py
+++ b/sdk/python/setup.py
@@ -50,6 +50,7 @@
     'kfp-pipeline-spec>=0.1.10,<0.2.0',
     'fire>=0.3.1,<1',
     'protobuf>=3.13.0,<4',
+    'uritemplate>=3.0.1,<4',
     # Standard library backports
     'dataclasses;python_version<"3.7"',
     'typing-extensions>=3.10.0.2,<4;python_version<"3.9"',


Test patch: