Repository: kubeflow/pipelines

Issue:
[Bug] Deploying 'metadata' from KFP manifests breaks KF Metadata
I'm testing the `metadata-writer` component locally, on an existing KF v0.7 (MiniKF on GCP).
First of all, it is a great service and looks pretty useful.

So, I deployed it using the manifests in this repo ([manifests/kustomize](https://github.com/kubeflow/pipelines/tree/0.1.40/manifests/kustomize)).

I found out that this installation _breaks_ the global `metadata` component.
The manifests modify metadata as follows:
* `metadata-deployment`: Instead of deploying a container similar to the existing one, that is an HTTP server, a gRPC server gets deployed there.
  Before:
  ```yaml
      containers:
      - command:
        - ./server/server
        - --http_port=8080
        - --mysql_service_host=metadata-db.kubeflow
        - --mysql_service_port=3306
        - --mysql_service_user=root
        - --mysql_service_password=$(MYSQL_ROOT_PASSWORD)
        - --mlmd_db_name=metadb
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              key: MYSQL_ROOT_PASSWORD
              name: metadata-db-secrets
        image: gcr.io/kubeflow-images-public/metadata:v0.1.10
        name: container
        ports:
        - containerPort: 8080
          name: backendapi
        readinessProbe:
          httpGet:
            httpHeaders:
            - name: ContentType
              value: application/json
            path: /api/v1alpha1/artifact_types
            port: backendapi
          initialDelaySeconds: 3
          periodSeconds: 5
          timeoutSeconds: 2
  ```
  After:
  ```yaml
      containers:
      - args:
        - --grpc_port=8080
        - --mysql_config_database=$(MYSQL_DATABASE)
        - --mysql_config_host=$(MYSQL_HOST)
        - --mysql_config_port=$(MYSQL_PORT)
        - --mysql_config_user=$(DBCONFIG_USER)
        - --mysql_config_password=$(DBCONFIG_PASSWORD)
        command:
        - /bin/metadata_store_server
        env:
        - name: DBCONFIG_USER
          valueFrom:
            secretKeyRef:
              key: username
              name: mysql-credential
        - name: DBCONFIG_PASSWORD
          valueFrom:
            secretKeyRef:
              key: password
              name: mysql-credential
        - name: MYSQL_DATABASE
          valueFrom:
            configMapKeyRef:
              key: MYSQL_DATABASE
              name: metadata-mysql-configmap
        - name: MYSQL_HOST
          valueFrom:
            configMapKeyRef:
              key: MYSQL_HOST
              name: metadata-mysql-configmap
        - name: MYSQL_PORT
          valueFrom:
            configMapKeyRef:
              key: MYSQL_PORT
              name: metadata-mysql-configmap
        image: gcr.io/tfx-oss-public/ml_metadata_store_server:0.14.0
        name: container
        ports:
        - containerPort: 8080
          name: md-backendapi
  ```
  * `metadata-service`: Port name changes, but I guess that's minor.
  * `metadata-envoy`: Different image tag, KF deploys `gcr.io/ml-pipeline/envoy:metadata-grpc` while these manifests use `gcr.io/ml-pipeline/envoy:initial`
  * Different host & credentials for mysql

The most important change that breaks Metadata is changing the backend server. Artifact Store expects an HTTP server, while Metadata Writer expects a gRPC server, under the same service (`metadata-service.kubeflow`).
https://github.com/kubeflow/pipelines/blob/0577fe10fdc3c57cd56ca3b65198c03636ae4520/backend/metadata_writer/src/metadata_helpers.py#L23-L30

---

So, I've got some questions, trying to find out what's the rationale behind this.

1. Why modify the `metadata-deployment` to be a gRPC server (or create is as a gRPC server), since `metadata-grpc-deployment` exists (or can be created instead) and `metadata-ui` (when that exists) speaks HTTP through `metadata-service`?
   `metadata-grpc-deployment` **is**, actually, almost the same deployment (there are some minor differences in its arguments but it feels like it's just naming stuff).

2. Why change the name of `metadata-service` port to `md-backendapi`? I think it would be best practice to either change `kubeflow/manifests`, or in this repo. But we shouldn't have them differ.

3. Why use directly the `ml_metadata` package, instead of `kubeflow.metadata`? After some [pretty shallow] checking, it seems like we could use the `kubeflow/metadata` SDK which also permits accessing `ml_metadata` objects and their methods.

4. Why use different credentials for the database?

---

Actually, I tried deploying just the `metadata-writer` component (`Deployment`, `Role`, `RoleBinding` and `ServiceAccount`) but I changed the aforementioned lines as follows, to use the gRPC service, and it works just great:
```python
def connect_to_mlmd() -> metadata_store.MetadataStore:
    metadata_service_host = os.environ.get('METADATA_GRPC_SERVICE_SERVICE_HOST', 'metadata-grpc-service')
    metadata_service_port = int(os.environ.get('METADATA_GRPC_SERVICE_SERVICE_PORT', 8080))

   mlmd_connection_config = metadata_store_pb2.MetadataStoreClientConfig(
        host=metadata_service_host,
        port=metadata_service_port,
    )
```

---

All in all, I think we should do some modifications making our manifests and `metadata-writer` code correspond to the main Metadata installation.

Any thoughts?

cc @Ark-kun, since it was mainly you who wrote Metadata Writer and its manifests.


Discussion:
/cc: @Ark-kun 
@elikatsis I only wrote the Metadata-Writer (and the Deployment, Role, RoleBinding and ServiceAccount manifests for it), so I'm not sure about the compatibility issues of the rest of the `makifests/kustomize/metadata` files. (Maybe I should have put the MW to the `pipelines` folder instead).

As far as I understand the problem, the issues is that the manifests in `kubeflow/manifests` and `kubeflow/pipeline/manifests` have diverged. The Metadata Writer just follows the metadata service configuration in `kubeflow/pipeline/manifests`.

I think @rmgogogo or @dushyanthsc try help synchronizing the manifests.
@Ark-kun as discussed, if we plan to support metadata-write in Kubeflow, we should have the manifest in Kubeflow/manifest with the right service and configurations. 

cc: @rmgogogo 
I've talked with @dushyanthsc. There are indeed two different manifest repos. There might be different versions of services between those. The Pipelines part of the kubeflow manifest repo is updated at some point between releases. The Metadata Writer manifests have not been ported to Kubeflow yet. We'll do that at some point before the next Kubeflow release.

>Why modify the metadata-deployment to be a gRPC server (or create is as a gRPC server), since metadata-grpc-deployment exists (or can be created instead) and metadata-ui (when that exists) speaks HTTP through metadata-service?

The reason given by @dushyanthsc is that KFP never had the HTTP metadata service, so the gRPC service was just named metadata-service. I wonder are there any problem if someone tries synchronizing the name.

>Actually, I tried deploying just the metadata-writer component (Deployment, Role, RoleBinding and ServiceAccount) but I changed the aforementioned lines as follows, to use the gRPC service, and it works just great:

Glad to hear that it worked for you.
Can you try setting `METADATA_SERVICE_SERVICE_HOST' to 'metadata-grpc-service` in the deployment spec container config, so that the MW code does not need to change?
@Ark-kun,
I'm all up for renaming the deployment and the service.
What I don't agree with is changing the values of `METADATA_SERVICE_*` variables.

The `kubelet` creates a set of variables for each service and makes each pod aware of them.

We shouldn't interfere with what K8s does, ignore, or override it. It's definitely not good practice.
So, I suggest we don't change the environment variables.

May I ask why is it you don't want the two-line change in `MetadataStore` initialization? It won't make any difference as far as the releases are concerned and it will be a clean fixup.
I can open a PR with all the required changes: renaming of resources & usage of the correct variable. It should be a pretty small one.

Another question you may missed: how come you didn't use the `kubeflow.metadata` SDK? No strong opinion here, I just think it nicely wraps Metadata entities.

@dushyanthsc @rmgogogo,
If it's of your interest, I've already done some work on adding `metadata-writer` in kubeflow/manifests. I haven't opened a PR for it yet but I'm just some testing before I verify it functions properly.
> May I ask why is it you don't want the two-line change in `MetadataStore` initialization? It won't make any difference as far as the releases are concerned and it will be a clean fixup.

This won't work with the current kustomize manifests as there is no `METADATA_GRPC_SERVICE_SERVICE`. I'm just the consumer of the existing metadata server deployer. My code uses what's available. Please check the manifest ownership.

>Another question you may missed: how come you didn't use the kubeflow.metadata SDK? No strong opinion here, I just think it nicely wraps Metadata entities.

I was mostly following the TFX handling of metadata. We can think about using a higher-level MLMD API. Even the MLMD itself now has a nicer higher-level API.
>Please check the manifest ownership.

I've seen the manifests, but you (the one who wrote those) assume a standalone installation.
I argue that the manifests should be able to perform both a standalone installation and a KFP upgrade in an existing KF deployment. And, of course, with no extra configuration or the breaking of some other KF component.

These manifests don't enable such thing. If you believe they shouldn't, I feel like it is something that we should openly discuss and put notes everywhere, because it breaks the whole (aka KF deployment), which is the most common case.

>This won't work with the current kustomize manifests as there is no `METADATA_GRPC_SERVICE_SERVICE`. I'm just the consumer of the existing metadata server deployer. My code uses what's available.

Of course I wouldn't change the `metadata-writer`'s code and the manifests in separate PRs. All should be done in a single PR and align this Metadata deployment with the main Metadata deployment. Nothing will break.
That is what i described here:
>I can open a PR with all the required changes: renaming of resources & usage of the correct variable. It should be a pretty small one.
> I've seen the manifests, but you (the one who wrote those) assume a standalone installation.

Yes. The MW code and deployer manifests are added to the standalone install manifests and that's what they assume.

KF installation is different, uses different versions, different manifests, different tools.
KF 0.7 ships with Pipelines v0.1.31 which does not have MW. That configuration works.https://github.com/kubeflow/manifests/tree/master/pipeline
When KF ships with Pipelines version >0.2, the MW should be there and working.

Installing manifests from `kubeflow/pipelines/manifests/kustomize/` over cluster created using the manifests from `kubeflow/manifests/metadata` is not supported.
The manifests in `kubeflow/pipelines/manifests/kustomize/` and `kubeflow/manifests/metadata` are maintained by different people with different priorities and different tools. It's unfortunate that these manifests diverge. I've raised this issue before and maybe in future there would be less divergence.

I'll leave it to the relevant people to discuss the synchronization of the non-MW manifests.
@dushyanthsc,

I understand you are the original commiter for these manifests ([PR: Switch to gRPC server](https://github.com/kubeflow/pipelines/pull/1973)). I was bitten by the problem of them having diverged from what KF ships, so in the interest of improving ease of use of current KFP with current KF, I'd be willing to commit the relevant changes, if it's OK with you. I understand the required changes are:
1. Rename `metadata-deployment` to `metadata-grpc-deployment`
2. Rename `metadata-service` to `metadata-grpc-service`
3. Make metadata writer's code correspond to this change, which means fixing the following lines https://github.com/kubeflow/pipelines/blob/0577fe10fdc3c57cd56ca3b65198c03636ae4520/backend/metadata_writer/src/metadata_helpers.py#L24-L25

@Ark-kun,

>Can you try setting `METADATA_SERVICE_SERVICE_HOST to metadata-grpc-service` in the deployment spec container config, so that the MW code does not need to change?

To summarize my argument: I understand that you ask to modify the environment variable's value, but this goes against the best practice of just using the variables that K8s provides ([docs](https://kubernetes.io/docs/concepts/services-networking/service/#environment-variables)).
Is it OK with you if I open a PR with the aforementioned changes?

>I'll leave it to the relevant people to discuss the synchronization of the non-MW manifests.

I am not sure I understand who the relevant people are, I contacted you and the rest of the team as you are the original committers of these manifests. Would it make sense to involve @jlewi in this?
@elikatsis @Ark-kun @rmgogogo 

I am don't have a strong opinion on the name of the deployment(`metadata-deployment` vs `metadata-grpc-deployment`)  and service(`metadata-service` vs `metadata-grpc-service`) here in kubeflow pipelines. There value also needs to change in https://github.com/kubeflow/pipelines/blob/master/manifests/kustomize/base/metadata/metadata-configmap.yaml and whereever it is used in https://github.com/kubeflow/pipelines/tree/master/manifests/kustomize

However, I do have strong opinion of whether the MW manifest should be expected to work across both kubeflow/pipelines and kubeflow/manifest repos. And they SHOULD NOT BE EXPECTED to work. As its pointed out earlier, they have different release cycles, testing support, manifest etc. We if need to introduce MW to kubeflow/manifest please introduce the MW yamls in the repo.

>they SHOULD NOT BE EXPECTED to work. As its pointed out earlier, they have different release cycles, testing support, manifest etc. 

I agree with Dushyanth.
Case in point: The oncoming change to MetadataWriter will make it incompatible with pre-0.2.0 Frontend versions (it will switch to writing different properties). So there is only a short window when MW is code-compatible with the Pipelines versions shipped in KF0.7.
@dushyanthsc,

>However, I do have strong opinion of whether the MW manifest should be expected to work across both kubeflow/pipelines and kubeflow/manifest repos. And they SHOULD NOT BE EXPECTED to work.

Ok, I understand that.
However, in this specific case, it is a matter of just a few of lines of change and we will have everything defined and used in a uniform way. It is also a matter of source code, not only manifests.
I also understand that two projects may diverge, but let's not diverge just for divergence sake when it is quite simple to make them compatible.

As @Ark-kun said,
>When KF ships with Pipelines version >0.2, the MW should be there and working.

Since these changes will be implemented, why not do it now? We will implement it after all.
Also, we will not release manifests contradicting to K8s best practices.

MW is a great service (kudos to Alexey). We know that users who use KF and run pipelines will find this feature very useful. They would love the `Artifacts/Executions` part in KFP UI filled, and they would definitely not want to break the main Metadata installation, which offers great UX as it is being developed.

>The oncoming change to MetadataWriter will make it incompatible with pre-0.2.0 Frontend versions (it will switch to writing different properties).

Yes, it is totally normal and expected for the MW to not function properly for with old UI versions, and that is the reason why manifests pin versions for every component. And only images from the same release are expected to work.

The changes I suggest don't really have to do with deploying a MW version different from the rest KFP components. Instead, they aim to KFP metadata manifests to correspond to KF metadata manifests (as they will sooner or later), and they will enable upgrading KFP in an existing KF cluster.

To summarize, I just propose renaming the metadata backend deployment and its service from `metadata-*` to `metadata-grpc-*`, and their usage. That very small change will make these manifests compatible with their KF counterparts.

@dushyanthsc, based on this:
>I am don't have a strong opinion on the name of the deployment(metadata-deployment vs metadata-grpc-deployment) and service(metadata-service vs metadata-grpc-service) here in kubeflow pipelines. [...]

Since you don't have strong opinion on this, let's go with the option that will maintain compatibility.
@elikatsis I 100%ly agree we should synchronize manifests naming.

One note is that, after renaming, we should mention in upgrade instructions to delete the resource with the old name.

Resolving patch:
diff --git a/backend/metadata_writer/src/metadata_helpers.py b/backend/metadata_writer/src/metadata_helpers.py
--- a/backend/metadata_writer/src/metadata_helpers.py
+++ b/backend/metadata_writer/src/metadata_helpers.py
@@ -21,8 +21,10 @@
 
 
 def connect_to_mlmd() -> metadata_store.MetadataStore:
-    metadata_service_host = os.environ.get('METADATA_SERVICE_SERVICE_HOST', 'metadata-service')
-    metadata_service_port = int(os.environ.get('METADATA_SERVICE_SERVICE_PORT', 8080))
+    metadata_service_host = os.environ.get(
+        'METADATA_GRPC_SERVICE_SERVICE_HOST', 'metadata-grpc-service')
+    metadata_service_port = int(os.environ.get(
+        'METADATA_GRPC_SERVICE_SERVICE_PORT', 8080))
 
     mlmd_connection_config = metadata_store_pb2.MetadataStoreClientConfig(
         host=metadata_service_host,


Test patch: