Repository: kubeflow/pipelines

Issue:
support separate pipeline for each namespace
EDIT from @Bobgy: This issue got 47 upvotes when requesting user feedback: https://github.com/kubeflow/pipelines/issues/1223#issuecomment-656508302.

### Proposed change
Currently with 1.0.2 version, Kubeflow Deployment with kfctl_k8s_istio shares pipelines for all namespaces defined in user profile. Separate pipeline support for each namespace is needed because the multi-user notebook server separation is already supported. It is natural to support pipeline separation.  

### Alternative options
NA

### Who would use this feature?
A lot of enterprises will benefit from this feature as allocating different namespaces to different teams is a common practice in Kubernetes and resources in existing namespaces can be effectively used.

### Suggest a solution
NA


Discussion:
FYI, in existing to be released KF 1.1, pipeline runs are already in user namespaces.

but the static pipeline yaml files are not separated. Do you want this additionally?
Related to https://github.com/kubeflow/pipelines/issues/1223
There are more clarification in https://github.com/kubeflow/pipelines/issues/1223#issuecomment-656507073.

If you agree with the proposal in https://github.com/kubeflow/pipelines/issues/1223#issuecomment-656508302, can you rephrase your description to just focus on pipeline resource for this issue.
I think the pipeline yaml files should be supported also for E2E separation.  Thanks. Already upvote in #1223 .
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

/frozen
Sorry, I'm kind of oversubscribed because of extra Kubeflow duties recently, so I may not be able to take this issue soon.

Leaving my previous thoughts about this:

For use-case context: some users want full separation of pipelines, and some want no separation. The distinction is mostly organization culture and I think both requests are valid.

So I think a MVP UX I can imagine that satisfies both is to:
1. pipelines in DB should have a namespace column
    * if namespace == '' (empty string), then it's in the shared space
    * if namespace is not empty, it's in that namespace
2. all requests to get pipelines can specify namespace='' to query only shared pipelines
3. or specify namespace='XXX' to find pipelines in a namespace + shared pipelines
4. We need to add a config option that disables shared pipelines for organizations that definitely do not want shared pipelines
5. We need to add a KFP UI option to switch between uploading a pipeline to shared space or current namespace.

Benefits:
1. The same backend logic seems to be able to support both 'all shared' or 'all separated' use cases.
2. There's no DB migration needed, this will be a seamless transition from older versions of KFP.

Probably there are cases I haven't fully thought through. I'd suggest anyone who's willing to push this issue forward to do the following:
1. try to discuss and converge on a set of CUJs (critical user journey) that works for everyone, note we should probably start with MVPs that address immediate concerns, but make sure we can head towards the direction that all valid use-cases can be supported.
2. after consensus on the UX, discuss with maintainers a rough design of the implementation
3. contribute PRs to implement this feature in smaller self-contained steps
/cc @yanniszark 
about this
and
/cc @chensun 
who designed multi user backend separation for other KFP resources
@Bobgy @chensun 

> We need to add a KFP UI option to switch between uploading a pipeline to shared space or current namespace.

Do we want user to upload their pipeline to share space? Should they use Managed Contributor instead like we did for experiments and runs? 

> There's no DB migration needed, this will be a seamless transition from older versions of KFP.

Pipeline schema doesn't have namespace concept yet. seamless migration here means implicitly migration, right? for all previous pipeline, we can think they have all empty namespace

> @Bobgy @chensun
> 
> > We need to add a KFP UI option to switch between uploading a pipeline to shared space or current namespace.
> 
> Do we want user to upload their pipeline to share space? Should they use Managed Contributor instead like we did for experiments and runs?

Some customers I worked with prefer this shared space though. A problem with manage contributor is that, they want to take a pipeline from shared space and run it in their own namespace. This will not be possible with adding contributors.

(It might be a good idea, to allow specifying pipelines in other users' namespace and KFP backend checks for that permission like contributors. Is that what you are proposing?)
 
> > There's no DB migration needed, this will be a seamless transition from older versions of KFP.
> 
> Pipeline schema doesn't have namespace concept yet. seamless migration here means implicitly migration, right? for all previous pipeline, we can think they have all empty namespace

If we do not introduce the shared pipeline concept, then we either need to figure out which pipeline belongs to which namespace or make all past pipelines inaccessible after an upgrade. This is fairly tricky to deal with.
> A problem with manage contributor is that, they want to take a pipeline from shared space and run it in their own namespace. This will not be possible with adding contributors.

Got it, they like to "folk" the pipeline into their own namespace. Managed contributor can only manage pipeline between users but not able to public to all. I think this makes sense. 

> to allow specifying pipelines in other users' namespace
It might be hard to deal with pipeline discovery (current KFP doesn't have granular control). User may have to provide pipeline name in other user's namespace. 

I think what you propose is a simple and better way to support isolated and shared pipelines. 


> then we either need to figure out which pipeline belongs to which namespace 

I notice UI actually determine namespace from centralboard. https://github.com/kubeflow/pipelines/blob/935a9b5ba5057bc9801fee87ed17c03c2907ec85/frontend/src/lib/KubeflowClient.tsx#L23-L33. 

Do you think it's better to figure out username and corresponding namespace via headers and KFAM?
backend can parse user's email and check kfam to figure out which namespace this profile owns and the namespace other users share with the current user.  Then KFP resources can be filtered by these namespaces. 

With this we can decouple centraldashboard from KFP. I think multi-user KFP can be used separated with Central dashboard. (probably only need KFAM profile and istio)

Regarding namespace selector, I think the topic was brought up before too, if we plan to let KFP support multi user mode without central dashboard, we can build a namespace selector similar to the one in centraldashboard. It won't be so much effort on the UI side (it was designed to be very decoupled in UI code from the beginning).

Regarding to KFAM, we have decided to deprecate it, @elikatsis has sent out a PR a few days ago: https://github.com/kubeflow/pipelines/pull/4723. Let's move related discussion to that PR, and keep discussion here focused on separating pipeline resources.
@Bobgy 
> 1. pipelines in DB should have a namespace column

Shall we have Namespace in pipeline_versions schema? For example I see Description is in pipelines but not in pipeline_versions. Not sure what is the criteria to decide here, but seems users are not allowed to edit or change pipelines once loaded, right?

> 4. We need to add a config option that disables shared pipelines for organizations that definitely do not want shared pipelines

A config option shall allow organizations to enable/disable the possibility pipelines can be public. Where do you think this config can be set (DB, configMap, config.json)?

> @Bobgy 
> > 1. pipelines in DB should have a namespace column
> 
> Shall we have Namespace in pipeline_versions schema? For example I see Description is in pipelines but not in pipeline_versions. Not sure what is the criteria to decide here, but seems users are not allowed to edit or change pipelines once loaded, right?

Versions are sub resources of pipelines, so they should be in the same namespace of the pipeline.

Description was more of an unfinished migration to version API, ideally we should have description field on versions too.

> > 4. We need to add a config option that disables shared pipelines for organizations that definitely do not want shared pipelines
> 
> A config option shall allow organizations to enable/disable the possibility pipelines can be public. Where do you think this config can be set (DB, configMap, config.json)?

You should use viper to get this config like other configs in API server, so that it will be configurable via config map, config.json or env var directly.


@Jeffwan @Bobgy 

We completed the implementation using KFam (for now). From a UI standpoint, we only added a checkbox to indicate if the Pipeline is "Shared" or not, Pipeline Versions of course cannot be shared. So if you select "create new pipeline..." this checkmark will not be there. 

Most of the changes were in the Backend. You can see the implementation details on this commit: https://github.com/arllanos/pipelines/commit/2c88722f6f05b67acc16c2b4e7bc54ff91c3f36c

We have not implemented an Env Variable to disable "Pipeline Sharing", however this shouldn't be too bad I think it should suffice to disable the button all together and Pipelines will be namespaced by default. 

Next week I'll switch the authentication to SubjectAccessReview and make the PR. 

One thing to note is that, "Shared Pipelines" are still deletable by other users in the System. I'm not sure what the ideal scenario should be here. 

![Screen Shot 2020-11-19 at 11 41 12 AM](https://user-images.githubusercontent.com/15258405/99703332-9af46d80-2a5c-11eb-9c3e-58654331863e.png)
@Bobdy @maganaluis thanks for your efforts on this. Some comments:

> or specify namespace='XXX' to find pipelines in a namespace + shared pipelines

@Bobgy I think it would be best to not return mixed results here. Specifying namespace='XXX' should only return pipelines from namespace 'XXX'.

> We need to add a config option that disables shared pipelines for organizations that definitely do not want shared pipelines

@Bobgy let me propose something different. Instead of turning off features, why not protect them with authorization? For example, we can have two kinds (in RBAC only, not in the MySQL DB), `Pipelines` and `ClusterPipelines` (namespace=="", name inspired from Role and ClusterRole). If an organization doesn't want to use one or the other, they simply need to not give their users permissions for one or the other.
> @bobdy @maganaluis thanks for your efforts on this. Some comments:
> 
> > or specify namespace='XXX' to find pipelines in a namespace + shared pipelines
> 
> @Bobgy I think it would be best to not return mixed results here. Specifying namespace='XXX' should only return pipelines from namespace 'XXX'.

You are right. Agree with that. So there needs to be some UI change to allow finding pipelines from either shared or namespaced ones.

> > We need to add a config option that disables shared pipelines for organizations that definitely do not want shared pipelines
> 
> @Bobgy let me propose something different. Instead of turning off features, why not protect them with authorization? For example, we can have two kinds (in RBAC only, not in the MySQL DB), `Pipelines` and `ClusterPipelines` (namespace=="", name inspired from Role and ClusterRole). If an organization doesn't want to use one or the other, they simply need to not give their users permissions for one or the other.

I guess it depends more on whether the shared pipelines will be a long term UX we will maintain. If it is, then I agree with your suggestion, that sounds like a clean solution. Do we have enough evidence it will be?

I think there're still quite some questions we need a clear answer on.
@maganaluis will you be willing to draft a rough design doc for this issue
Hmm, maybe we can start with your PR description for discussion and see if everyone can agree on that.

Note that, for above discussion, my personal opinion is that, the MVP PR do not need to implement any of the `ClusterPipeline` rbac stuff nor the "pipeline sharing" switch. We can start from the minimal and enhance based on further requests.
> One thing to note is that, "Shared Pipelines" are still deletable by other users in the System. I'm not sure what the ideal scenario should be here.

It's backward compatible behavior, so I don't think we need to worry about that right now.

Resolving patch:
diff --git a/backend/api/python_http_client/kfp_server_api/api/pipeline_service_api.py b/backend/api/python_http_client/kfp_server_api/api/pipeline_service_api.py
--- a/backend/api/python_http_client/kfp_server_api/api/pipeline_service_api.py
+++ b/backend/api/python_http_client/kfp_server_api/api/pipeline_service_api.py
@@ -1190,6 +1190,10 @@ def list_pipelines(self, **kwargs):  # noqa: E501
         :type sort_by: str
         :param filter: A url-encoded, JSON-serialized Filter protocol buffer (see [filter.proto](https://github.com/kubeflow/pipelines/ blob/master/backend/api/filter.proto)).
         :type filter: str
+        :param resource_reference_key_type: The type of the resource that referred to.
+        :type resource_reference_key_type: str
+        :param resource_reference_key_id: The ID of the resource that referred to.
+        :type resource_reference_key_id: str
         :param async_req: Whether to execute the request asynchronously.
         :type async_req: bool, optional
         :param _preload_content: if False, the urllib3.HTTPResponse object will
@@ -1224,6 +1228,10 @@ def list_pipelines_with_http_info(self, **kwargs):  # noqa: E501
         :type sort_by: str
         :param filter: A url-encoded, JSON-serialized Filter protocol buffer (see [filter.proto](https://github.com/kubeflow/pipelines/ blob/master/backend/api/filter.proto)).
         :type filter: str
+        :param resource_reference_key_type: The type of the resource that referred to.
+        :type resource_reference_key_type: str
+        :param resource_reference_key_id: The ID of the resource that referred to.
+        :type resource_reference_key_id: str
         :param async_req: Whether to execute the request asynchronously.
         :type async_req: bool, optional
         :param _return_http_data_only: response data without head status code
@@ -1249,7 +1257,9 @@ def list_pipelines_with_http_info(self, **kwargs):  # noqa: E501
             'page_token',
             'page_size',
             'sort_by',
-            'filter'
+            'filter',
+            'resource_reference_key_type',
+            'resource_reference_key_id'
         ]
         all_params.extend(
             [
@@ -1282,6 +1292,10 @@ def list_pipelines_with_http_info(self, **kwargs):  # noqa: E501
             query_params.append(('sort_by', local_var_params['sort_by']))  # noqa: E501
         if 'filter' in local_var_params and local_var_params['filter'] is not None:  # noqa: E501
             query_params.append(('filter', local_var_params['filter']))  # noqa: E501
+        if 'resource_reference_key_type' in local_var_params and local_var_params['resource_reference_key_type'] is not None:  # noqa: E501
+            query_params.append(('resource_reference_key.type', local_var_params['resource_reference_key_type']))  # noqa: E501
+        if 'resource_reference_key_id' in local_var_params and local_var_params['resource_reference_key_id'] is not None:  # noqa: E501
+            query_params.append(('resource_reference_key.id', local_var_params['resource_reference_key_id']))  # noqa: E501
 
         header_params = {}
 
diff --git a/backend/api/python_http_client/kfp_server_api/models/api_pipeline.py b/backend/api/python_http_client/kfp_server_api/models/api_pipeline.py
--- a/backend/api/python_http_client/kfp_server_api/models/api_pipeline.py
+++ b/backend/api/python_http_client/kfp_server_api/models/api_pipeline.py
@@ -40,7 +40,8 @@ class ApiPipeline(object):
         'parameters': 'list[ApiParameter]',
         'url': 'ApiUrl',
         'error': 'str',
-        'default_version': 'ApiPipelineVersion'
+        'default_version': 'ApiPipelineVersion',
+        'resource_references': 'list[ApiResourceReference]'
     }
 
     attribute_map = {
@@ -51,10 +52,11 @@ class ApiPipeline(object):
         'parameters': 'parameters',
         'url': 'url',
         'error': 'error',
-        'default_version': 'default_version'
+        'default_version': 'default_version',
+        'resource_references': 'resource_references'
     }
 
-    def __init__(self, id=None, created_at=None, name=None, description=None, parameters=None, url=None, error=None, default_version=None, local_vars_configuration=None):  # noqa: E501
+    def __init__(self, id=None, created_at=None, name=None, description=None, parameters=None, url=None, error=None, default_version=None, resource_references=None, local_vars_configuration=None):  # noqa: E501
         """ApiPipeline - a model defined in OpenAPI"""  # noqa: E501
         if local_vars_configuration is None:
             local_vars_configuration = Configuration()
@@ -68,6 +70,7 @@ def __init__(self, id=None, created_at=None, name=None, description=None, parame
         self._url = None
         self._error = None
         self._default_version = None
+        self._resource_references = None
         self.discriminator = None
 
         if id is not None:
@@ -86,6 +89,8 @@ def __init__(self, id=None, created_at=None, name=None, description=None, parame
             self.error = error
         if default_version is not None:
             self.default_version = default_version
+        if resource_references is not None:
+            self.resource_references = resource_references
 
     @property
     def id(self):
@@ -267,6 +272,29 @@ def default_version(self, default_version):
 
         self._default_version = default_version
 
+    @property
+    def resource_references(self):
+        """Gets the resource_references of this ApiPipeline.  # noqa: E501
+
+        Input. Required. E.g., specify which pipeline this pipeline version belongs to.  # noqa: E501
+
+        :return: The resource_references of this ApiPipeline.  # noqa: E501
+        :rtype: list[ApiResourceReference]
+        """
+        return self._resource_references
+
+    @resource_references.setter
+    def resource_references(self, resource_references):
+        """Sets the resource_references of this ApiPipeline.
+
+        Input. Required. E.g., specify which pipeline this pipeline version belongs to.  # noqa: E501
+
+        :param resource_references: The resource_references of this ApiPipeline.  # noqa: E501
+        :type resource_references: list[ApiResourceReference]
+        """
+
+        self._resource_references = resource_references
+
     def to_dict(self):
         """Returns the model properties as a dict"""
         result = {}


Test patch:
diff --git a/backend/api/python_http_client/test/test_api_list_pipelines_response.py b/backend/api/python_http_client/test/test_api_list_pipelines_response.py
--- a/backend/api/python_http_client/test/test_api_list_pipelines_response.py
+++ b/backend/api/python_http_client/test/test_api_list_pipelines_response.py
@@ -64,7 +64,11 @@ def make_instance(self, include_optional):
                                         id = '0', ), 
                                     name = '0', 
                                     relationship = 'UNKNOWN_RELATIONSHIP', )
-                                ], ), )
+                                ], ), 
+                        resource_references = [
+                            kfp_server_api.models.api_resource_reference.apiResourceReference(
+                                name = '0', )
+                            ], )
                     ], 
                 total_size = 56, 
                 next_page_token = '0'
diff --git a/backend/api/python_http_client/test/test_api_pipeline.py b/backend/api/python_http_client/test/test_api_pipeline.py
--- a/backend/api/python_http_client/test/test_api_pipeline.py
+++ b/backend/api/python_http_client/test/test_api_pipeline.py
@@ -67,7 +67,15 @@ def make_instance(self, include_optional):
                                 id = '0', ), 
                             name = '0', 
                             relationship = 'UNKNOWN_RELATIONSHIP', )
-                        ], )
+                        ], ), 
+                resource_references = [
+                    kfp_server_api.models.api_resource_reference.apiResourceReference(
+                        key = kfp_server_api.models.api_resource_key.apiResourceKey(
+                            type = 'UNKNOWN_RESOURCE_TYPE', 
+                            id = '0', ), 
+                        name = '0', 
+                        relationship = 'UNKNOWN_RELATIONSHIP', )
+                    ]
             )
         else :
             return ApiPipeline(
diff --git a/backend/src/apiserver/resource/resource_manager_test.go b/backend/src/apiserver/resource/resource_manager_test.go
--- a/backend/src/apiserver/resource/resource_manager_test.go
+++ b/backend/src/apiserver/resource/resource_manager_test.go
@@ -79,7 +79,7 @@ func initWithPipeline(t *testing.T) (*FakeClientManager, *ResourceManager, *mode
 	initEnvVars()
 	store := NewFakeClientManagerOrFatal(util.NewFakeTimeForEpoch())
 	manager := NewResourceManager(store)
-	p, err := manager.CreatePipeline("p1", "", []byte(testWorkflow.ToStringForStore()))
+	p, err := manager.CreatePipeline("p1", "", "", []byte(testWorkflow.ToStringForStore()))
 	assert.Nil(t, err)
 	return store, manager, p
 }
@@ -101,7 +101,7 @@ func initWithExperimentAndPipeline(t *testing.T) (*FakeClientManager, *ResourceM
 	apiExperiment := &api.Experiment{Name: "e1"}
 	experiment, err := manager.CreateExperiment(apiExperiment)
 	assert.Nil(t, err)
-	pipeline, err := manager.CreatePipeline("p1", "", []byte(testWorkflow.ToStringForStore()))
+	pipeline, err := manager.CreatePipeline("p1", "", "", []byte(testWorkflow.ToStringForStore()))
 	assert.Nil(t, err)
 	return store, manager, experiment, pipeline
 }
@@ -225,7 +225,8 @@ func TestCreatePipeline(t *testing.T) {
 			Parameters:     "[{\"name\":\"param1\"}]",
 			Status:         model.PipelineVersionReady,
 			PipelineId:     DefaultFakeUUID,
-		}}
+		},
+		Namespace: ""}
 	assert.Equal(t, pipelineExpected, pipeline)
 }
 
@@ -234,7 +235,7 @@ func TestCreatePipeline_ComplexPipeline(t *testing.T) {
 	defer store.Close()
 	manager := NewResourceManager(store)
 
-	createdPipeline, err := manager.CreatePipeline("pipeline1", "", []byte(strings.TrimSpace(
+	createdPipeline, err := manager.CreatePipeline("pipeline1", "", "", []byte(strings.TrimSpace(
 		complexPipeline)))
 	assert.Nil(t, err)
 	_, err = manager.GetPipeline(createdPipeline.UUID)
@@ -245,7 +246,7 @@ func TestCreatePipeline_GetParametersError(t *testing.T) {
 	store := NewFakeClientManagerOrFatal(util.NewFakeTimeForEpoch())
 	defer store.Close()
 	manager := NewResourceManager(store)
-	_, err := manager.CreatePipeline("pipeline1", "", []byte("I am invalid yaml"))
+	_, err := manager.CreatePipeline("pipeline1", "", "", []byte("I am invalid yaml"))
 	assert.Equal(t, codes.InvalidArgument, err.(*util.UserError).ExternalStatusCode())
 	assert.Contains(t, err.Error(), "Failed to parse the parameter")
 }
@@ -255,7 +256,7 @@ func TestCreatePipeline_StorePipelineMetadataError(t *testing.T) {
 	defer store.Close()
 	store.DB().Close()
 	manager := NewResourceManager(store)
-	_, err := manager.CreatePipeline("pipeline1", "", []byte("apiVersion: argoproj.io/v1alpha1\nkind: Workflow"))
+	_, err := manager.CreatePipeline("pipeline1", "", "", []byte("apiVersion: argoproj.io/v1alpha1\nkind: Workflow"))
 	assert.Equal(t, codes.Internal, err.(*util.UserError).ExternalStatusCode())
 	assert.Contains(t, err.Error(), "Failed to start a transaction to create a new pipeline")
 }
@@ -266,7 +267,7 @@ func TestCreatePipeline_CreatePipelineFileError(t *testing.T) {
 	manager := NewResourceManager(store)
 	// Use a bad object store
 	manager.objectStore = &FakeBadObjectStore{}
-	_, err := manager.CreatePipeline("pipeline1", "", []byte("apiVersion: argoproj.io/v1alpha1\nkind: Workflow"))
+	_, err := manager.CreatePipeline("pipeline1", "", "", []byte("apiVersion: argoproj.io/v1alpha1\nkind: Workflow"))
 	assert.Equal(t, codes.Internal, err.(*util.UserError).ExternalStatusCode())
 	assert.Contains(t, err.Error(), "bad object store")
 	// Verify there is a pipeline in DB with status PipelineCreating.
@@ -1879,7 +1880,7 @@ func TestReportScheduledWorkflowResource_Error(t *testing.T) {
 	workflow := util.NewWorkflow(&v1alpha1.Workflow{
 		TypeMeta:   v1.TypeMeta{APIVersion: "argoproj.io/v1alpha1", Kind: "Workflow"},
 		ObjectMeta: v1.ObjectMeta{Name: "workflow-name"}})
-	p, err := manager.CreatePipeline("1", "", []byte(workflow.ToStringForStore()))
+	p, err := manager.CreatePipeline("1", "", "", []byte(workflow.ToStringForStore()))
 	assert.Nil(t, err)
 
 	// Create job
@@ -2388,7 +2389,7 @@ func TestCreatePipelineVersion(t *testing.T) {
 	manager := NewResourceManager(store)
 
 	// Create a pipeline before versions.
-	_, err := manager.CreatePipeline("p", "", []byte(testWorkflow.ToStringForStore()))
+	_, err := manager.CreatePipeline("p", "", "", []byte(testWorkflow.ToStringForStore()))
 	assert.Nil(t, err)
 
 	// Create a version under the above pipeline.
@@ -2429,7 +2430,7 @@ func TestCreatePipelineVersion_ComplexPipelineVersion(t *testing.T) {
 	manager := NewResourceManager(store)
 
 	// Create a pipeline.
-	createdPipeline, err := manager.CreatePipeline("pipeline", "", []byte(strings.TrimSpace(complexPipeline)))
+	createdPipeline, err := manager.CreatePipeline("pipeline", "", "", []byte(strings.TrimSpace(complexPipeline)))
 	assert.Nil(t, err)
 
 	// Create a version under the above pipeline.
@@ -2465,7 +2466,7 @@ func TestCreatePipelineVersion_CreatePipelineVersionFileError(t *testing.T) {
 	manager := NewResourceManager(store)
 
 	// Create a pipeline.
-	_, err := manager.CreatePipeline("pipeline", "", []byte(strings.TrimSpace(complexPipeline)))
+	_, err := manager.CreatePipeline("pipeline", "", "", []byte(strings.TrimSpace(complexPipeline)))
 	assert.Nil(t, err)
 
 	// Switch to a bad object store
@@ -2504,7 +2505,7 @@ func TestCreatePipelineVersion_GetParametersError(t *testing.T) {
 	manager := NewResourceManager(store)
 
 	// Create a pipeline.
-	_, err := manager.CreatePipeline("pipeline", "", []byte(testWorkflow.ToStringForStore()))
+	_, err := manager.CreatePipeline("pipeline", "", "", []byte(testWorkflow.ToStringForStore()))
 	assert.Nil(t, err)
 
 	// Create a version under the above pipeline.
@@ -2538,6 +2539,7 @@ func TestCreatePipelineVersion_StorePipelineVersionMetadataError(t *testing.T) {
 	_, err := manager.CreatePipeline(
 		"pipeline",
 		"",
+		"",
 		[]byte("apiVersion: argoproj.io/v1alpha1\nkind: Workflow"))
 	assert.Nil(t, err)
 
@@ -2574,7 +2576,7 @@ func TestDeletePipelineVersion(t *testing.T) {
 	manager := NewResourceManager(store)
 
 	// Create a pipeline.
-	_, err := manager.CreatePipeline("pipeline", "", []byte("apiVersion: argoproj.io/v1alpha1\nkind: Workflow"))
+	_, err := manager.CreatePipeline("pipeline", "", "", []byte("apiVersion: argoproj.io/v1alpha1\nkind: Workflow"))
 	assert.Nil(t, err)
 
 	// Create a version under the above pipeline.
@@ -2611,7 +2613,7 @@ func TestDeletePipelineVersion_FileError(t *testing.T) {
 	manager := NewResourceManager(store)
 
 	// Create a pipeline.
-	_, err := manager.CreatePipeline("pipeline", "", []byte("apiVersion: argoproj.io/v1alpha1\nkind: Workflow"))
+	_, err := manager.CreatePipeline("pipeline", "", "", []byte("apiVersion: argoproj.io/v1alpha1\nkind: Workflow"))
 	assert.Nil(t, err)
 
 	// Create a version under the above pipeline.
diff --git a/backend/src/apiserver/server/pipeline_server_test.go b/backend/src/apiserver/server/pipeline_server_test.go
--- a/backend/src/apiserver/server/pipeline_server_test.go
+++ b/backend/src/apiserver/server/pipeline_server_test.go
@@ -263,6 +263,26 @@ func TestListPipelineVersion_NoResourceKey(t *testing.T) {
 	assert.Equal(t, "Invalid input error: ResourceKey must be set in the input", err.Error())
 }
 
+func TestListPipelinesPublic(t *testing.T) {
+	httpServer := getMockServer(t)
+	// Close the server when test finishes
+	defer httpServer.Close()
+	clientManager := resource.NewFakeClientManagerOrFatal(util.NewFakeTimeForEpoch())
+	resourceManager := resource.NewResourceManager(clientManager)
+
+	pipelineServer := PipelineServer{resourceManager: resourceManager, httpClient: httpServer.Client(), options: &PipelineServerOptions{CollectMetrics: false}}
+	_, err := pipelineServer.ListPipelines(context.Background(),
+		&api.ListPipelinesRequest{
+			PageSize: 20,
+			ResourceReferenceKey: &api.ResourceKey{
+				Type: api.ResourceType_NAMESPACE,
+				Id:   "",
+			},
+		})
+	assert.EqualValues(t, nil, err, err)
+
+}
+
 func TestCreatePipelineVersionDontUpdateDefault(t *testing.T) {
 	viper.Set(common.UpdatePipelineVersionByDefault, "false")
 	defer viper.Set(common.UpdatePipelineVersionByDefault, "true")
diff --git a/backend/src/apiserver/server/pipeline_upload_server_test.go b/backend/src/apiserver/server/pipeline_upload_server_test.go
--- a/backend/src/apiserver/server/pipeline_upload_server_test.go
+++ b/backend/src/apiserver/server/pipeline_upload_server_test.go
@@ -85,7 +85,7 @@ func TestUploadPipeline_YAML(t *testing.T) {
 				Status:         model.PipelineVersionReady,
 				PipelineId:     resource.DefaultFakeUUID,
 			}}}
-	pkg, totalSize, str, err := clientManager.PipelineStore().ListPipelines(opts)
+	pkg, totalSize, str, err := clientManager.PipelineStore().ListPipelines(&common.FilterContext{}, opts)
 	assert.Nil(t, err)
 	assert.Equal(t, str, "")
 	assert.Equal(t, 1, totalSize)
@@ -171,7 +171,7 @@ func TestUploadPipeline_Tarball(t *testing.T) {
 				Status:         model.PipelineVersionReady,
 				PipelineId:     resource.DefaultFakeUUID,
 			}}}
-	pkg, totalSize, str, err := clientManager.PipelineStore().ListPipelines(opts)
+	pkg, totalSize, str, err := clientManager.PipelineStore().ListPipelines(&common.FilterContext{}, opts)
 	assert.Nil(t, err)
 	assert.Equal(t, str, "")
 	assert.Equal(t, 1, totalSize)
@@ -267,7 +267,7 @@ func TestUploadPipeline_SpecifyFileName(t *testing.T) {
 				Status:         model.PipelineVersionReady,
 				PipelineId:     resource.DefaultFakeUUID,
 			}}}
-	pkg, totalSize, str, err := clientManager.PipelineStore().ListPipelines(opts)
+	pkg, totalSize, str, err := clientManager.PipelineStore().ListPipelines(&common.FilterContext{}, opts)
 	assert.Nil(t, err)
 	assert.Equal(t, 1, totalSize)
 	assert.Equal(t, str, "")
@@ -322,7 +322,7 @@ func TestUploadPipeline_SpecifyFileDescription(t *testing.T) {
 			},
 			Description: "description of foo bar",
 		}}
-	pkg, totalSize, str, err := clientManager.PipelineStore().ListPipelines(opts)
+	pkg, totalSize, str, err := clientManager.PipelineStore().ListPipelines(&common.FilterContext{}, opts)
 	assert.Nil(t, err)
 	assert.Equal(t, 1, totalSize)
 	assert.Equal(t, str, "")
diff --git a/backend/src/apiserver/server/test_util.go b/backend/src/apiserver/server/test_util.go
--- a/backend/src/apiserver/server/test_util.go
+++ b/backend/src/apiserver/server/test_util.go
@@ -160,7 +160,7 @@ func initWithExperimentAndPipelineVersion(t *testing.T) (*resource.FakeClientMan
 	assert.Nil(t, err)
 
 	// Create a pipeline and then a pipeline version.
-	_, err = resourceManager.CreatePipeline("pipeline", "", []byte("apiVersion: argoproj.io/v1alpha1\nkind: Workflow"))
+	_, err = resourceManager.CreatePipeline("pipeline", "", "", []byte("apiVersion: argoproj.io/v1alpha1\nkind: Workflow"))
 	assert.Nil(t, err)
 	_, err = resourceManager.CreatePipelineVersion(&api.PipelineVersion{
 		Name: "pipeline_version",
@@ -190,7 +190,7 @@ func initWithExperimentsAndTwoPipelineVersions(t *testing.T) (*resource.FakeClie
 	assert.Nil(t, err)
 
 	// Create a pipeline and then a pipeline version.
-	_, err = resourceManager.CreatePipeline("pipeline", "", []byte("apiVersion: argoproj.io/v1alpha1\nkind: Workflow"))
+	_, err = resourceManager.CreatePipeline("pipeline", "", "", []byte("apiVersion: argoproj.io/v1alpha1\nkind: Workflow"))
 	assert.Nil(t, err)
 	_, err = resourceManager.CreatePipelineVersion(&api.PipelineVersion{
 		Name: "pipeline_version",
@@ -209,9 +209,9 @@ func initWithExperimentsAndTwoPipelineVersions(t *testing.T) (*resource.FakeClie
 	clientManager.UpdateUUID(util.NewFakeUUIDGeneratorOrFatal(resource.NonDefaultFakeUUID, nil))
 	resourceManager = resource.NewResourceManager(clientManager)
 	// Create another pipeline and then pipeline version.
-	_, err = resourceManager.CreatePipeline("anpther-pipeline", "", []byte("apiVersion: argoproj.io/v1alpha1\nkind: Workflow"))
+	_, err = resourceManager.CreatePipeline("anpther-pipeline", "", "", []byte("apiVersion: argoproj.io/v1alpha1\nkind: Workflow"))
 	assert.Nil(t, err)
-	_, err =  resourceManager.CreatePipelineVersion(&api.PipelineVersion{
+	_, err = resourceManager.CreatePipelineVersion(&api.PipelineVersion{
 		Name: "another_pipeline_version",
 		ResourceReferences: []*api.ResourceReference{
 			&api.ResourceReference{
@@ -254,7 +254,7 @@ func initWithPipeline(t *testing.T) (*resource.FakeClientManager, *resource.Reso
 	initEnvVars()
 	store := resource.NewFakeClientManagerOrFatal(util.NewFakeTimeForEpoch())
 	manager := resource.NewResourceManager(store)
-	p, err := manager.CreatePipeline("p1", "", []byte(testWorkflow.ToStringForStore()))
+	p, err := manager.CreatePipeline("p1", "", "", []byte(testWorkflow.ToStringForStore()))
 	assert.Nil(t, err)
 	return store, manager, p
 }
diff --git a/backend/src/apiserver/storage/pipeline_store_test.go b/backend/src/apiserver/storage/pipeline_store_test.go
--- a/backend/src/apiserver/storage/pipeline_store_test.go
+++ b/backend/src/apiserver/storage/pipeline_store_test.go
@@ -18,6 +18,7 @@ import (
 	"testing"
 
 	api "github.com/kubeflow/pipelines/backend/api/go_client"
+	"github.com/kubeflow/pipelines/backend/src/apiserver/common"
 	"github.com/kubeflow/pipelines/backend/src/apiserver/list"
 	"github.com/kubeflow/pipelines/backend/src/apiserver/model"
 	"github.com/kubeflow/pipelines/backend/src/common/util"
@@ -95,7 +96,7 @@ func TestListPipelines_FilterOutNotReady(t *testing.T) {
 	opts, err := list.NewOptions(&model.Pipeline{}, 10, "id", nil)
 	assert.Nil(t, err)
 
-	pipelines, total_size, nextPageToken, err := pipelineStore.ListPipelines(opts)
+	pipelines, total_size, nextPageToken, err := pipelineStore.ListPipelines(&common.FilterContext{}, opts)
 
 	assert.Nil(t, err)
 	assert.Equal(t, "", nextPageToken)
@@ -141,7 +142,7 @@ func TestListPipelines_WithFilter(t *testing.T) {
 	opts, err := list.NewOptions(&model.Pipeline{}, 10, "id", filterProto)
 	assert.Nil(t, err)
 
-	pipelines, totalSize, nextPageToken, err := pipelineStore.ListPipelines(opts)
+	pipelines, totalSize, nextPageToken, err := pipelineStore.ListPipelines(&common.FilterContext{}, opts)
 
 	assert.Nil(t, err)
 	assert.Equal(t, "", nextPageToken)
@@ -194,7 +195,7 @@ func TestListPipelines_Pagination(t *testing.T) {
 
 	opts, err := list.NewOptions(&model.Pipeline{}, 2, "name", nil)
 	assert.Nil(t, err)
-	pipelines, total_size, nextPageToken, err := pipelineStore.ListPipelines(opts)
+	pipelines, total_size, nextPageToken, err := pipelineStore.ListPipelines(&common.FilterContext{}, opts)
 	assert.Nil(t, err)
 	assert.NotEmpty(t, nextPageToken)
 	assert.Equal(t, 4, total_size)
@@ -235,7 +236,7 @@ func TestListPipelines_Pagination(t *testing.T) {
 	opts, err = list.NewOptionsFromToken(nextPageToken, 2)
 	assert.Nil(t, err)
 
-	pipelines, total_size, nextPageToken, err = pipelineStore.ListPipelines(opts)
+	pipelines, total_size, nextPageToken, err = pipelineStore.ListPipelines(&common.FilterContext{}, opts)
 	assert.Nil(t, err)
 	assert.Empty(t, nextPageToken)
 	assert.Equal(t, 4, total_size)
@@ -288,7 +289,7 @@ func TestListPipelines_Pagination_Descend(t *testing.T) {
 
 	opts, err := list.NewOptions(&model.Pipeline{}, 2, "name desc", nil)
 	assert.Nil(t, err)
-	pipelines, total_size, nextPageToken, err := pipelineStore.ListPipelines(opts)
+	pipelines, total_size, nextPageToken, err := pipelineStore.ListPipelines(&common.FilterContext{}, opts)
 	assert.Nil(t, err)
 	assert.NotEmpty(t, nextPageToken)
 	assert.Equal(t, 4, total_size)
@@ -328,7 +329,7 @@ func TestListPipelines_Pagination_Descend(t *testing.T) {
 
 	opts, err = list.NewOptionsFromToken(nextPageToken, 2)
 	assert.Nil(t, err)
-	pipelines, total_size, nextPageToken, err = pipelineStore.ListPipelines(opts)
+	pipelines, total_size, nextPageToken, err = pipelineStore.ListPipelines(&common.FilterContext{}, opts)
 	assert.Nil(t, err)
 	assert.Empty(t, nextPageToken)
 	assert.Equal(t, 4, total_size)
@@ -359,7 +360,7 @@ func TestListPipelines_Pagination_LessThanPageSize(t *testing.T) {
 
 	opts, err := list.NewOptions(&model.Pipeline{}, 2, "", nil)
 	assert.Nil(t, err)
-	pipelines, total_size, nextPageToken, err := pipelineStore.ListPipelines(opts)
+	pipelines, total_size, nextPageToken, err := pipelineStore.ListPipelines(&common.FilterContext{}, opts)
 	assert.Nil(t, err)
 	assert.Equal(t, "", nextPageToken)
 	assert.Equal(t, 1, total_size)
@@ -373,7 +374,7 @@ func TestListPipelinesError(t *testing.T) {
 	db.Close()
 	opts, err := list.NewOptions(&model.Pipeline{}, 2, "", nil)
 	assert.Nil(t, err)
-	_, _, _, err = pipelineStore.ListPipelines(opts)
+	_, _, _, err = pipelineStore.ListPipelines(&common.FilterContext{}, opts)
 	assert.Equal(t, codes.Internal, err.(*util.UserError).ExternalStatusCode())
 }
 
diff --git a/backend/test/integration/pipeline_api_test.go b/backend/test/integration/pipeline_api_test.go
--- a/backend/test/integration/pipeline_api_test.go
+++ b/backend/test/integration/pipeline_api_test.go
@@ -28,6 +28,7 @@ import (
 // - Providing tarball file url
 type PipelineApiTest struct {
 	suite.Suite
+	namespace            string
 	pipelineClient       *api_server.PipelineClient
 	pipelineUploadClient *api_server.PipelineUploadClient
 }
@@ -45,6 +46,7 @@ func (s *PipelineApiTest) SetupTest() {
 			glog.Exitf("Failed to initialize test. Error: %s", err.Error())
 		}
 	}
+	s.namespace = *namespace
 	clientConfig := test.GetClientConfig(*namespace)
 	var err error
 	s.pipelineUploadClient, err = api_server.NewPipelineUploadClient(clientConfig, false)
@@ -98,7 +100,7 @@ func (s *PipelineApiTest) TestPipelineAPI() {
 	assert.Equal(t, "arguments.pipeline.zip", argumentUrlPipeline.Name)
 
 	/* ---------- Verify list pipeline works ---------- */
-	pipelines, totalSize, _, err := s.pipelineClient.List(params.NewListPipelinesParams())
+	pipelines, totalSize, _, err := s.pipelineClient.List(&params.ListPipelinesParams{})
 	assert.Nil(t, err)
 	assert.Equal(t, 4, len(pipelines))
 	assert.Equal(t, 4, totalSize)
diff --git a/backend/test/integration/pipeline_version_api_test.go b/backend/test/integration/pipeline_version_api_test.go
--- a/backend/test/integration/pipeline_version_api_test.go
+++ b/backend/test/integration/pipeline_version_api_test.go
@@ -70,7 +70,7 @@ func (s *PipelineVersionApiTest) TestPipelineVersionAPI() {
 	assert.Equal(t, "test_pipeline", pipeline.Name)
 
 	/* ---------- Get pipeline id ---------- */
-	pipelines, totalSize, _, err := s.pipelineClient.List(params.NewListPipelinesParams())
+	pipelines, totalSize, _, err := s.pipelineClient.List(&params.ListPipelinesParams{})
 	assert.Nil(t, err)
 	assert.Equal(t, 1, len(pipelines))
 	assert.Equal(t, 1, totalSize)