Repository: kubeflow/pipelines

Issue:
[FR] Implement optional component level usage tracking
As #3578 checked in, we have an example of how to set beacon to collect critical usage information to help us make the right business decision, without compromising user privacy.

One natural extension of that work is to instrument telemetry of our first-party components shipped with KFP. To be concrete, the proposed scope includes components under [1]. We can consider whether/how to include user-defined component later on.

Still my proposed approach will be facilitated by pod labels append to each ContainerOp.

1. We establish a whitelisting dictionary, only including first-party components; I am considering using the component url as signature.
2. In [2], when loading a component, we'll append telemetry pod labels, which will then surfaced in GKE.
3. We provide toggle to allow users to opt out this collection, at both component loading, and compiler.

[1] https://github.com/kubeflow/pipelines/tree/master/components

[2] https://github.com/kubeflow/pipelines/blob/master/sdk/python/kfp/components/_components.py


Any input will be very helpful :) @Ark-kun @rmgogogo 



Discussion:
I want to note that the SDK already adds the `pipelines.kubeflow.org/component_spec` pod annotation for every component. This information includes the whole component definition, but does not contain the URI.
Adding `pipelines.kubeflow.org/component_ref` annotation that often contains the `.uri` attribute would be pretty trivial and also useful for UX.

P.S. Adding `pipelines.kubeflow.org/task_spec` is also an option. It will contain both `component_ref` and `arguments` which will fix the issue with input parameters experienced by the Metadata Writer.

>at both component loading

I'm not sure this is necessary. The annotation of pods is performed by the compiler, not the component loader.
> I want to note that the SDK already adds the pipelines.kubeflow.org/component_spec pod annotation

Thanks.IIRC we only have pod annotation for those information, right? For technical reason I think adding corresponding pod labels will be necessary. Do you see any problem with adding a pod label including part of the information in component_spec?
> Do you see any problem with adding a pod label including part of the information in component_spec?

We can try to add some label, but it's not trivial. You cannot really put, say, a URL in a label. Or anything substantial. The size is limited to 63 mostly alphanumerical characters (so even some hashes do not fit). https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set

>Valid label values must be 63 characters or less and must be empty or begin and end with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), underscores (_), dots (.), and alphanumerics between.
Thanks. I think we shall be smart about what to put in that label. Let me think about how to better leverage the 63 character budget.
> Thanks. I think we shall be smart about what to put in that label. Let me think about how to better leverage the 63 character budget.

Technically, every component has a natural digest (SHA256 of the serialized ComponentSpec that was loaded from file or URL). SHA256 fits in the label. But associating it with a particular component version would be non-trivial.

We can also base-64 encode the component URL if it's not too long.
qq: here it all for KFP SDK's component, right?
@rmgogogo  Yes that's what I am thinking
Every pod created from a component now has the `component_ref` annotation in addition to the full `component_spec` annotation (which included component name and inputs/outputs). Added in https://github.com/kubeflow/pipelines/pull/3727 and https://github.com/kubeflow/pipelines/pull/3726
```
annotations:
    pipelines.kubeflow.org/component_ref: '{
        "digest": "98381958ba8b0d2b83a23a78f482f08b48e665409820b3a6254bccdbcf206df3",
        "url": "https://raw.githubusercontent.com/kubeflow/pipelines/b3179d86b239a08bf4884b50dbf3a9151da96d66/components/gcp/automl/create_dataset_for_tables/component.yaml"
    }'

```
Thanks @Ark-kun . However we still need pod labels to do something similar because pod annotations do not persist in the internal GKE log.

Resolving patch:
diff --git a/sdk/python/kfp/compiler/_default_transformers.py b/sdk/python/kfp/compiler/_default_transformers.py
--- a/sdk/python/kfp/compiler/_default_transformers.py
+++ b/sdk/python/kfp/compiler/_default_transformers.py
@@ -12,7 +12,9 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import re
 from typing import Callable, Dict, Optional, Text
+
 from ..dsl._container_op import BaseOp, ContainerOp
 
 # Pod label indicating the SDK type from which the pipeline is
@@ -20,6 +22,16 @@
 _SDK_ENV_LABEL = 'pipelines.kubeflow.org/pipeline-sdk-type'
 _SDK_ENV_DEFAULT = 'kfp'
 
+# Common prefix of KFP OOB components url paths.
+_OOB_COMPONENT_PATH_PREFIX = 'https://raw.githubusercontent.com/kubeflow/'\
+                             'pipelines'
+
+# Key for component origin path pod label.
+COMPONENT_PATH_LABEL_KEY = 'pipelines.kubeflow.org/component_origin_path'
+
+# Key for component spec digest pod label.
+COMPONENT_DIGEST_LABEL_KEY = 'pipelines.kubeflow.org/component_digest'
+
 
 def get_default_telemetry_labels() -> Dict[Text, Text]:
     """Returns the default pod labels for telemetry purpose."""
@@ -68,3 +80,40 @@ def _add_pod_labels(task):
         return task
 
     return _add_pod_labels
+
+
+def _remove_suffix(string: Text, suffix: Text) -> Text:
+    """Removes the suffix from a string."""
+    if suffix and string.endswith(suffix):
+        return string[:-len(suffix)]
+    else:
+        return string
+
+
+def add_name_for_oob_components() -> Callable:
+    """Adds the OOB component name if applicable."""
+    
+    def _add_name_for_oob_components(task):
+        # Detect the component origin uri in component_ref if exists, and
+        # attach the OOB component name as a pod label.
+        component_ref = getattr(task, '_component_ref', None)
+        if component_ref:
+            if component_ref.url:
+                origin_path = _remove_suffix(
+                    component_ref.url, 'component.yaml').rstrip('/')
+                # Only include KFP OOB components.
+                if origin_path.startswith(_OOB_COMPONENT_PATH_PREFIX):
+                    origin_path = origin_path.split('/', 7)[-1]
+                else:
+                    return task
+                # Clean the label to comply with the k8s label convention.
+                origin_path = re.sub('[^-a-z0-9A-Z_.]', '.', origin_path)
+                origin_path_label = origin_path[-63:].strip('-_.')
+                task.add_pod_label(COMPONENT_PATH_LABEL_KEY, origin_path_label)
+            if component_ref.digest:
+                task.add_pod_label(
+                    COMPONENT_DIGEST_LABEL_KEY, component_ref.digest)
+            
+        return task
+    
+    return _add_name_for_oob_components
\ No newline at end of file
diff --git a/sdk/python/kfp/compiler/compiler.py b/sdk/python/kfp/compiler/compiler.py
--- a/sdk/python/kfp/compiler/compiler.py
+++ b/sdk/python/kfp/compiler/compiler.py
@@ -27,7 +27,7 @@
 from .. import dsl
 from ._k8s_helper import convert_k8s_obj_to_json, sanitize_k8s_name
 from ._op_to_template import _op_to_template
-from ._default_transformers import add_pod_env, add_pod_labels, get_default_telemetry_labels
+from ._default_transformers import add_pod_env, add_pod_labels, add_name_for_oob_components, get_default_telemetry_labels
 
 from ..components.structures import InputSpec
 from ..components._yaml_utils import dump_yaml
@@ -836,6 +836,7 @@ def _create_workflow(self,
     if allow_telemetry:
       pod_labels = get_default_telemetry_labels()
       op_transformers.append(add_pod_labels(pod_labels))
+      op_transformers.append(add_name_for_oob_components())
 
     op_transformers.extend(pipeline_conf.op_transformers)
 


Test patch:
diff --git a/sdk/python/tests/compiler/compiler_tests.py b/sdk/python/tests/compiler/compiler_tests.py
--- a/sdk/python/tests/compiler/compiler_tests.py
+++ b/sdk/python/tests/compiler/compiler_tests.py
@@ -26,6 +26,8 @@
 import unittest
 import yaml
 
+from kfp import components
+from kfp.compiler._default_transformers import COMPONENT_DIGEST_LABEL_KEY, COMPONENT_PATH_LABEL_KEY
 from kfp.dsl._component import component
 from kfp.dsl import ContainerOp, pipeline
 from kfp.dsl.types import Integer, InconsistentTypeException
@@ -40,6 +42,11 @@ def some_op():
       command=['sleep 1'],
   )
 
+_TEST_GCS_DOWNLOAD_COMPONENT_URL = 'https://raw.githubusercontent.com/kubeflow/'\
+                                   'pipelines/2dac60c400ad8767b452649d08f328df'\
+                                   'af230f96/components/google-cloud/storage/'\
+                                   'download/component.yaml'
+
 
 class TestCompiler(unittest.TestCase):
   # Define the places of samples covered by unit tests.
@@ -711,6 +718,27 @@ def some_pipeline():
       container = template.get('container', None)
       if container:
         self.assertEqual(template['retryStrategy']['limit'], 5)
+        
+  def test_oob_component_label(self):
+    gcs_download_op = components.load_component_from_url(
+        _TEST_GCS_DOWNLOAD_COMPONENT_URL)
+    
+    @dsl.pipeline(name='some_pipeline')
+    def some_pipeline():
+      _download_task = gcs_download_op('gs://some_bucket/some_dir/some_file')
+      
+    workflow_dict = compiler.Compiler()._compile(some_pipeline)
+    
+    found_download_task = False
+    for template in workflow_dict['spec']['templates']:
+      if template.get('container', None):
+        found_download_task = True
+        self.assertEqual(
+            template['metadata']['labels'][COMPONENT_PATH_LABEL_KEY],
+            'google-cloud.storage.download')
+        self.assertIsNotNone(
+            template['metadata']['labels'].get(COMPONENT_DIGEST_LABEL_KEY))
+    self.assertTrue(found_download_task, 'download task not found in workflow.')
   
   def test_image_pull_policy(self):
     def some_op():