Repository: kubeflow/pipelines

Issue:
[feature] support customizing Tensorboard images
### Feature Area

<!-- Uncomment the labels below which are relevant to this feature: -->
/area frontend
<!-- /area backend -->
<!-- /area sdk -->
<!-- /area samples -->
<!-- /area components -->


### What feature would you like to see?

<!-- Provide a description of this feature and the user experience. -->
Allow users to customize the tensorboard image they want to use in KFP component visualizations.
https://www.kubeflow.org/docs/components/pipelines/sdk/output-viewer/

There are mainly two ways I can think of to support this feature:

Option 1:
1. add an input textbox for image in Tensorboard plugin that users can directly edit
2. (optional) let the textbox remember historic entries

Option 2 - let component output decide
1. when a component emits visualization metadata, add a new field `image` they can specify tensorboard image
2. KFP UI loads the `mlpipeline-ui-metadata` and default to show the image option in component metadata

### What is the use case or pain point?

<!-- It helps us understand the benefit of this feature for your use case. -->

Existing issues that can be addressed by this feature: https://github.com/kubeflow/pipelines/issues/5449

It's currently not possible to choose any tensorboard image version unless it is [hard-coded in KFP UI](https://github.com/kubeflow/pipelines/blob/2c682299202ad019bb2b2bb2ac8903c72ab245cf/frontend/src/components/viewers/Tensorboard.tsx#L216).

Users or pipeline components know the best about which tensorboard version/image type can visualize its output, not the KFP cluster operator.

### Is there a workaround currently?

<!-- Without this feature, how do you accomplish your task today? -->
No

---

<!-- Don't delete message below to encourage users to support your feature request! -->
Love this idea? Give it a 👍. We prioritize fulfilling features with the most 👍.

[Tensorboard] Allow custom env vars per instance
From https://github.com/kubeflow/pipelines/issues/3936#issuecomment-721529801, there are use cases when it makes sense to let users configure arguments a certain tensorboard instance should have.

I haven't thought over this much, welcome contribution on brainstorming how the UX should like and contribute it to KFP.


Discussion:
I think option 1 and 2 can probably both be useful
Some feature requests that is related to this https://github.com/kubeflow/pipelines/issues/1641 https://github.com/kubeflow/pipelines/issues/4714
Note, a prerequisite for implementing this feature is supporting passing full image to KFP viewer API. Currently we can only pass tensorboard version there: see https://github.com/kubeflow/pipelines/blob/2c682299202ad019bb2b2bb2ac8903c72ab245cf/frontend/src/components/viewers/Tensorboard.tsx#L215-L229 and https://github.com/kubeflow/pipelines/blob/2c682299202ad019bb2b2bb2ac8903c72ab245cf/frontend/server/handlers/tensorboard.ts#L69
@Bobgy Being able to pass full image paths is a must for most on-prem installations since they use an internal docker registry. For instance, our k8s klusters have no external access and any tensorflow-based image needs our internal certificates added.
/assign
I will work on this shortly, the best way to help is confirm whether above suggested options fit your use-case.

I will implement option 2 directly, because that seems like the fundamental requirement.
@Bobgy This seems like a good option.
@Bobgy Thanks, btw which version can I expect it to land on?

@lechwolowski 1.5.0 has been too close to a final release, so you should expect it to land on 1.6.0, but I can share a prebuilt image you can put in your cluster first to get this feature. KFP UI server has been very stable, so I won't expect much incompatibilities.
FYI, we already support customizing tensorboards via a static patch e.g. https://github.com/kubeflow/pipelines/blob/master/docs/config/volume-support.md#deploy-ml-pipeline-ui-to-support-tensorboard-viewer.

This issue is about defining a UX that allows dynamic configurations.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

/lifecycle frozen
Quotes from https://github.com/kubeflow/pipelines/issues/5426.
It's important to support several env vars that can be different per pipeline for using tensorboard with S3.

From @ConverJens:
> @Bobgy @shrinath-suresh Generally, to use anything Tensorflow-based (e.g. TF serving, TFX) the following environment variables needs to be set:
> ENDPOINT_URL
> S3_USE_HTTPS
> S3_VERIFY_SSL
> AWS_ACCESS_KEY_ID
> AWS_SECRET_ACCESS_KEY
> 
> Additionally, one might need to add own certificates if you're self-hosting your S3 solution (e.g. Minio, Ceph) on-prem.

From @shrinath-suresh
> @Bobgy We have identified the solution. Tensorboard with S3 buckets works based on region. The bucket which we used was in different region. Once we updated the AWS_REGION variable in cluster, we are able to see the tensorboard UI

So, it's reasonable to add fields in tensorboard mlpipeline-ui-metadata spec ability to specify these environment variables.
See https://blog.min.io/hyper-scale-machine-learning-with-minio-and-tensorflow/,

> AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin AWS_REGION=us-east-1 S3_ENDPOINT=localhost:9000 S3_USE_HTTPS=0 S3_VERIFY_SSL=0 tensorboard --logdir s3://datasets/logs


The requirements for getting tensorboard to work with MinIO is similar to S3.
Very similar. We are currently using TFX and Minio in the fashion described in the post. The main difference is that Minio is usually self-hosted and hence, may require custom certificates.
Hi @ConverJens, thank you for sharing your scenario. What customizations do we need from KFP side to support custom certificate?
@Bobgy No problem. For us, it's sufficient if we can specify a custom tensorboard image and said env vars.

Resolving patch:
diff --git a/samples/core/visualization/tensorboard_gcs.py b/samples/core/visualization/tensorboard_gcs.py
new file mode 100644
--- /dev/null
+++ b/samples/core/visualization/tensorboard_gcs.py
@@ -0,0 +1,73 @@
+# Copyright 2021 The Kubeflow Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from kfp import dsl, components
+from kfp.components import create_component_from_func
+
+prepare_tensorboard = components.load_component_from_url(
+    'https://raw.githubusercontent.com/kubeflow/pipelines/1.5.0/components/tensorflow/tensorboard/prepare_tensorboard/component.yaml'
+)
+
+
+def train(log_dir: 'URI'):
+    # Reference: https://www.tensorflow.org/tensorboard/get_started
+    import tensorflow as tf
+
+    mnist = tf.keras.datasets.mnist
+
+    (x_train, y_train), (x_test, y_test) = mnist.load_data()
+    x_train, x_test = x_train / 255.0, x_test / 255.0
+
+    def create_model():
+        return tf.keras.models.Sequential([
+            tf.keras.layers.Flatten(input_shape=(28, 28)),
+            tf.keras.layers.Dense(512, activation='relu'),
+            tf.keras.layers.Dropout(0.2),
+            tf.keras.layers.Dense(10, activation='softmax')
+        ])
+
+    model = create_model()
+    model.compile(
+        optimizer='adam',
+        loss='sparse_categorical_crossentropy',
+        metrics=['accuracy']
+    )
+
+    tensorboard_callback = tf.keras.callbacks.TensorBoard(
+        log_dir=log_dir, histogram_freq=1
+    )
+
+    model.fit(
+        x=x_train,
+        y=y_train,
+        epochs=5,
+        validation_data=(x_test, y_test),
+        callbacks=[tensorboard_callback]
+    )
+
+
+# Be careful when choosing a tensorboard image:
+# * tensorflow/tensorflow may fail with image pull backoff, because of dockerhub rate limiting.
+# * tensorboard in tensorflow 2.3+ does not work with KFP, refer to https://github.com/kubeflow/pipelines/issues/5521.
+train_op = create_component_from_func(
+    train, base_image='gcr.io/deeplearning-platform-release/tf2-cpu.2-4'
+)
+
+
+@dsl.pipeline(name='pipeline-tensorboard-gcs')
+def my_pipeline(
+    log_dir=f'gs://{{kfp-default-bucket}}/tensorboard/logs/{dsl.RUN_ID_PLACEHOLDER}'
+):
+    prepare_tb_task = prepare_tensorboard(log_dir_uri=log_dir)
+    tensorboard_task = train_op(log_dir=prepare_tb_task.outputs['log_dir_uri'],)
diff --git a/samples/core/visualization/tensorboard_minio.py b/samples/core/visualization/tensorboard_minio.py
new file mode 100644
--- /dev/null
+++ b/samples/core/visualization/tensorboard_minio.py
@@ -0,0 +1,184 @@
+# Copyright 2021 The Kubeflow Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import json
+from kfp.onprem import use_k8s_secret
+from kfp import dsl, components
+from kfp.components import OutputPath, create_component_from_func
+
+prepare_tensorboard = components.load_component_from_file(
+    os.path.join(
+        os.path.dirname(__file__),
+        '../../../components/tensorflow/tensorboard/prepare_tensorboard/component.yaml'
+    )
+)
+# New features used locally not released yet, so we can only use local import.
+# prepare_tensorboard = components.load_component_from_url(
+#     'https://raw.githubusercontent.com/kubeflow/pipelines/1.5.0/components/tensorflow/tensorboard/prepare_tensorboard/component.yaml'
+# )
+
+
+def train(minio_endpoint: 'URI', log_bucket: str, log_dir: 'Path'):
+    # Reference: https://www.tensorflow.org/tensorboard/get_started
+    import tensorflow as tf
+    import datetime
+
+    mnist = tf.keras.datasets.mnist
+
+    (x_train, y_train), (x_test, y_test) = mnist.load_data()
+    x_train, x_test = x_train / 255.0, x_test / 255.0
+
+    def create_model():
+        return tf.keras.models.Sequential([
+            tf.keras.layers.Flatten(input_shape=(28, 28)),
+            tf.keras.layers.Dense(512, activation='relu'),
+            tf.keras.layers.Dropout(0.2),
+            tf.keras.layers.Dense(10, activation='softmax')
+        ])
+
+    model = create_model()
+    model.compile(
+        optimizer='adam',
+        loss='sparse_categorical_crossentropy',
+        metrics=['accuracy']
+    )
+
+    log_dir_local = "logs/fit"
+    tensorboard_callback = tf.keras.callbacks.TensorBoard(
+        log_dir=log_dir_local, histogram_freq=1
+    )
+
+    model.fit(
+        x=x_train,
+        y=y_train,
+        epochs=5,
+        validation_data=(x_test, y_test),
+        callbacks=[tensorboard_callback]
+    )
+
+    # Copy the local logs folder to minio.
+    #
+    # TODO: we may write a filesystem watch process that continuously copy logs
+    # dir to minio, so that we can watch live training logs via tensorboard.
+    #
+    # Note, although tensorflow supports minio via s3:// protocol. We want to
+    # demo how minio can be used instead, e.g. the same approach can be used with
+    # frameworks only support local path.
+    from minio import Minio
+    import os
+    minio_access_key = os.getenv('MINIO_ACCESS_KEY')
+    minio_secret_key = os.getenv('MINIO_SECRET_KEY')
+    if not minio_access_key or not minio_secret_key:
+        raise Exception('MINIO_ACCESS_KEY or MINIO_SECRET_KEY env is not set')
+    client = Minio(
+        minio_endpoint,
+        access_key=minio_access_key,
+        secret_key=minio_secret_key,
+        secure=False
+    )
+    count = 0
+    from pathlib import Path
+    for path in Path("logs").rglob("*"):
+        if not path.is_dir():
+            object_name = os.path.join(
+                log_dir, os.path.relpath(start=log_dir_local, path=path)
+            )
+            client.fput_object(
+                bucket_name=log_bucket,
+                object_name=object_name,
+                file_path=path,
+            )
+            count = count + 1
+            print(f'{path} uploaded to minio://{log_bucket}/{object_name}')
+    print(f'{count} log files uploaded to minio://{log_bucket}/{log_dir}')
+
+
+# tensorflow/tensorflow:2.4 may fail with image pull backoff, because of dockerhub rate limiting.
+train_op = create_component_from_func(
+    train,
+    base_image='gcr.io/deeplearning-platform-release/tf2-cpu.2-3:latest',
+    packages_to_install=['minio'],  # TODO: pin minio version
+)
+
+
+@dsl.pipeline(name='pipeline-tensorboard-minio')
+def my_pipeline(
+    minio_endpoint='minio-service:9000',
+    log_bucket='mlpipeline',
+    log_dir=f'tensorboard/logs/{dsl.RUN_ID_PLACEHOLDER}',
+    # Pin to tensorflow 2.3, because in 2.4+ tensorboard cannot load in KFP:
+    # refer to https://github.com/kubeflow/pipelines/issues/5521.
+    tf_image='gcr.io/deeplearning-platform-release/tf2-cpu.2-3:latest'
+):
+    # tensorboard uses s3 protocol to access minio
+    prepare_tb_task = prepare_tensorboard(
+        log_dir_uri=f's3://{log_bucket}/{log_dir}',
+        image=tf_image,
+        pod_template_spec=json.dumps({
+            'spec': {
+                'containers': [{
+                    # These env vars make tensorboard access KFP in-cluster minio
+                    # using s3 protocol.
+                    # Reference: https://blog.min.io/hyper-scale-machine-learning-with-minio-and-tensorflow/
+                    'env': [{
+                        'name': 'AWS_ACCESS_KEY_ID',
+                        'valueFrom': {
+                            'secretKeyRef': {
+                                'name': 'mlpipeline-minio-artifact',
+                                'key': 'accesskey'
+                            }
+                        }
+                    }, {
+                        'name': 'AWS_SECRET_ACCESS_KEY',
+                        'valueFrom': {
+                            'secretKeyRef': {
+                                'name': 'mlpipeline-minio-artifact',
+                                'key': 'secretkey'
+                            }
+                        }
+                    }, {
+                        'name': 'AWS_REGION',
+                        'value': 'minio'
+                    }, {
+                        'name': 'S3_ENDPOINT',
+                        'value': f'{minio_endpoint}',
+                    }, {
+                        'name': 'S3_USE_HTTPS',
+                        'value': '0',
+                    }, {
+                        'name': 'S3_VERIFY_SSL',
+                        'value': '0',
+                    }]
+                }],
+            },
+        })
+    )
+    train_task = train_op(
+        minio_endpoint=minio_endpoint,
+        log_bucket=log_bucket,
+        log_dir=log_dir,
+    )
+    train_task.apply(
+        use_k8s_secret(
+            secret_name='mlpipeline-minio-artifact',
+            k8s_secret_key_to_env={
+                'secretkey': 'MINIO_SECRET_KEY',
+                'accesskey': 'MINIO_ACCESS_KEY'
+            },
+        )
+    )
+    # optional, let training task use the same tensorflow image as specified tensorboard
+    train_task.container.image = tf_image
+    train_task.after(prepare_tb_task)


Test patch:
diff --git a/frontend/server/app.test.ts b/frontend/server/app.test.ts
--- a/frontend/server/app.test.ts
+++ b/frontend/server/app.test.ts
@@ -21,9 +21,6 @@ import { loadConfigs } from './configs';
 import { TEST_ONLY as K8S_TEST_EXPORT } from './k8s-helper';
 import { Server } from 'http';
 import { commonSetup } from './integration-tests/test-helper';
-import * as fs from 'fs';
-import * as path from 'path';
-import * as os from 'os';
 
 jest.mock('node-fetch');
 
@@ -367,770 +364,6 @@ describe('UIServer apis', () => {
     });
   });
 
-  describe('/apps/tensorboard', () => {
-    const POD_TEMPLATE_SPEC = {
-      spec: {
-        containers: [
-          {
-            volumeMounts: [
-              {
-                name: 'tensorboard',
-                mountPath: '/logs',
-              },
-              {
-                name: 'data',
-                subPath: 'tensorboard',
-                mountPath: '/data',
-              },
-            ],
-          },
-        ],
-        volumes: [
-          {
-            name: 'tensorboard',
-            persistentVolumeClaim: {
-              claimName: 'logs',
-            },
-          },
-          {
-            name: 'data',
-            persistentVolumeClaim: {
-              claimName: 'data',
-            },
-          },
-        ],
-      },
-    };
-
-    let k8sGetCustomObjectSpy: jest.SpyInstance;
-    let k8sDeleteCustomObjectSpy: jest.SpyInstance;
-    let k8sCreateCustomObjectSpy: jest.SpyInstance;
-    let kfpApiServer: Server;
-
-    function newGetTensorboardResponse({
-      name = 'viewer-example',
-      logDir = 'log-dir-example',
-      tensorflowImage = 'tensorflow:2.0.0',
-      type = 'tensorboard',
-    }: {
-      name?: string;
-      logDir?: string;
-      tensorflowImage?: string;
-      type?: string;
-    } = {}) {
-      return {
-        response: undefined as any, // unused
-        body: {
-          metadata: {
-            name,
-          },
-          spec: {
-            tensorboardSpec: { logDir, tensorflowImage },
-            type,
-          },
-        },
-      };
-    }
-
-    beforeEach(() => {
-      k8sGetCustomObjectSpy = jest.spyOn(
-        K8S_TEST_EXPORT.k8sV1CustomObjectClient,
-        'getNamespacedCustomObject',
-      );
-      k8sDeleteCustomObjectSpy = jest.spyOn(
-        K8S_TEST_EXPORT.k8sV1CustomObjectClient,
-        'deleteNamespacedCustomObject',
-      );
-      k8sCreateCustomObjectSpy = jest.spyOn(
-        K8S_TEST_EXPORT.k8sV1CustomObjectClient,
-        'createNamespacedCustomObject',
-      );
-    });
-
-    afterEach(() => {
-      if (kfpApiServer) {
-        kfpApiServer.close();
-      }
-    });
-
-    describe('get', () => {
-      it('requires logdir for get tensorboard', done => {
-        app = new UIServer(loadConfigs(argv, {}));
-        requests(app.start())
-          .get('/apps/tensorboard')
-          .expect(400, 'logdir argument is required', done);
-      });
-
-      it('requires namespace for get tensorboard', done => {
-        app = new UIServer(loadConfigs(argv, {}));
-        requests(app.start())
-          .get('/apps/tensorboard?logdir=some-log-dir')
-          .expect(400, 'namespace argument is required', done);
-      });
-
-      it('does not crash with a weird query', done => {
-        app = new UIServer(loadConfigs(argv, {}));
-        k8sGetCustomObjectSpy.mockImplementation(() =>
-          Promise.resolve(newGetTensorboardResponse()),
-        );
-        // The special case is that, decodeURIComponent('%2') throws an
-        // exception, so this can verify handler doesn't do extra
-        // decodeURIComponent on queries.
-        const weirdLogDir = encodeURIComponent('%2');
-        requests(app.start())
-          .get(`/apps/tensorboard?logdir=${weirdLogDir}&namespace=test-ns`)
-          .expect(200, done);
-      });
-
-      function setupMockKfpApiService({ port = 3001 }: { port?: number } = {}) {
-        const receivedHeaders: any[] = [];
-        kfpApiServer = express()
-          .get('/apis/v1beta1/auth', (req, res) => {
-            receivedHeaders.push(req.headers);
-            res.status(200).send('{}'); // Authorized
-          })
-          .listen(port);
-        return { receivedHeaders, host: 'localhost', port };
-      }
-
-      it('authorizes user requests from KFP auth api', done => {
-        const { receivedHeaders, host, port } = setupMockKfpApiService();
-        app = new UIServer(
-          loadConfigs(argv, {
-            ENABLE_AUTHZ: 'true',
-            ML_PIPELINE_SERVICE_PORT: `${port}`,
-            ML_PIPELINE_SERVICE_HOST: host,
-          }),
-        );
-        k8sGetCustomObjectSpy.mockImplementation(() =>
-          Promise.resolve(newGetTensorboardResponse()),
-        );
-        requests(app.start())
-          .get(`/apps/tensorboard?logdir=some-log-dir&namespace=test-ns`)
-          .set('x-goog-authenticated-user-email', 'accounts.google.com:user@google.com')
-          .expect(200, err => {
-            expect(receivedHeaders).toHaveLength(1);
-            expect(receivedHeaders[0]).toMatchInlineSnapshot(`
-              Object {
-                "accept": "*/*",
-                "accept-encoding": "gzip,deflate",
-                "connection": "close",
-                "host": "localhost:3001",
-                "user-agent": "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)",
-                "x-goog-authenticated-user-email": "accounts.google.com:user@google.com",
-              }
-            `);
-            done(err);
-          });
-      });
-
-      it('uses configured KUBEFLOW_USERID_HEADER for user identity', done => {
-        const { receivedHeaders, host, port } = setupMockKfpApiService();
-        app = new UIServer(
-          loadConfigs(argv, {
-            ENABLE_AUTHZ: 'true',
-            KUBEFLOW_USERID_HEADER: 'x-kubeflow-userid',
-            ML_PIPELINE_SERVICE_PORT: `${port}`,
-            ML_PIPELINE_SERVICE_HOST: host,
-          }),
-        );
-        k8sGetCustomObjectSpy.mockImplementation(() =>
-          Promise.resolve(newGetTensorboardResponse()),
-        );
-        requests(app.start())
-          .get(`/apps/tensorboard?logdir=some-log-dir&namespace=test-ns`)
-          .set('x-kubeflow-userid', 'user@kubeflow.org')
-          .expect(200, err => {
-            expect(receivedHeaders).toHaveLength(1);
-            expect(receivedHeaders[0]).toHaveProperty('x-kubeflow-userid', 'user@kubeflow.org');
-            done(err);
-          });
-      });
-
-      it('rejects user requests when KFP auth api rejected', done => {
-        const errorSpy = jest.spyOn(console, 'error');
-        errorSpy.mockImplementation();
-
-        const apiServerPort = 3001;
-        kfpApiServer = express()
-          .get('/apis/v1beta1/auth', (_, res) => {
-            res.status(400).send(
-              JSON.stringify({
-                error: 'User xxx is not unauthorized to list viewers',
-                details: ['unauthorized', 'callstack'],
-              }),
-            );
-          })
-          .listen(apiServerPort);
-        app = new UIServer(
-          loadConfigs(argv, {
-            ENABLE_AUTHZ: 'true',
-            ML_PIPELINE_SERVICE_PORT: `${apiServerPort}`,
-            ML_PIPELINE_SERVICE_HOST: 'localhost',
-          }),
-        );
-        k8sGetCustomObjectSpy.mockImplementation(() =>
-          Promise.resolve(newGetTensorboardResponse()),
-        );
-        requests(app.start())
-          .get(`/apps/tensorboard?logdir=some-log-dir&namespace=test-ns`)
-          .expect(
-            401,
-            'User is not authorized to GET VIEWERS in namespace test-ns: User xxx is not unauthorized to list viewers',
-            err => {
-              expect(errorSpy).toHaveBeenCalledTimes(1);
-              expect(
-                errorSpy,
-              ).toHaveBeenCalledWith(
-                'User is not authorized to GET VIEWERS in namespace test-ns: User xxx is not unauthorized to list viewers',
-                ['unauthorized', 'callstack'],
-              );
-              done(err);
-            },
-          );
-      });
-
-      it('gets tensorboard url and version', done => {
-        app = new UIServer(loadConfigs(argv, {}));
-        k8sGetCustomObjectSpy.mockImplementation(() =>
-          Promise.resolve(
-            newGetTensorboardResponse({
-              name: 'viewer-abcdefg',
-              logDir: 'log-dir-1',
-              tensorflowImage: 'tensorflow:2.0.0',
-            }),
-          ),
-        );
-
-        requests(app.start())
-          .get(`/apps/tensorboard?logdir=${encodeURIComponent('log-dir-1')}&namespace=test-ns`)
-          .expect(
-            200,
-            JSON.stringify({
-              podAddress:
-                'http://viewer-abcdefg-service.test-ns.svc.cluster.local:80/tensorboard/viewer-abcdefg/',
-              tfVersion: '2.0.0',
-            }),
-            err => {
-              expect(k8sGetCustomObjectSpy.mock.calls[0]).toMatchInlineSnapshot(`
-                              Array [
-                                "kubeflow.org",
-                                "v1beta1",
-                                "test-ns",
-                                "viewers",
-                                "viewer-5e1404e679e27b0f0b8ecee8fe515830eaa736c5",
-                              ]
-                          `);
-              done(err);
-            },
-          );
-      });
-    });
-
-    describe('post (create)', () => {
-      it('requires logdir', done => {
-        app = new UIServer(loadConfigs(argv, {}));
-        requests(app.start())
-          .post('/apps/tensorboard')
-          .expect(400, 'logdir argument is required', done);
-      });
-
-      it('requires namespace', done => {
-        app = new UIServer(loadConfigs(argv, {}));
-        requests(app.start())
-          .post('/apps/tensorboard?logdir=some-log-dir')
-          .expect(400, 'namespace argument is required', done);
-      });
-
-      it('requires tfversion', done => {
-        app = new UIServer(loadConfigs(argv, {}));
-        requests(app.start())
-          .post('/apps/tensorboard?logdir=some-log-dir&namespace=test-ns')
-          .expect(400, 'tfversion (tensorflow version) argument is required', done);
-      });
-
-      it('creates tensorboard viewer custom object and waits for it', done => {
-        let getRequestCount = 0;
-        k8sGetCustomObjectSpy.mockImplementation(() => {
-          ++getRequestCount;
-          switch (getRequestCount) {
-            case 1:
-              return Promise.reject('Not found');
-            case 2:
-              return Promise.resolve(
-                newGetTensorboardResponse({
-                  name: 'viewer-abcdefg',
-                  logDir: 'log-dir-1',
-                  tensorflowImage: 'tensorflow:2.0.0',
-                }),
-              );
-            default:
-              throw new Error('only expected to be called twice in this test');
-          }
-        });
-        k8sCreateCustomObjectSpy.mockImplementation(() => Promise.resolve());
-
-        app = new UIServer(loadConfigs(argv, {}));
-        requests(app.start())
-          .post(
-            `/apps/tensorboard?logdir=${encodeURIComponent(
-              'log-dir-1',
-            )}&namespace=test-ns&tfversion=2.0.0`,
-          )
-          .expect(
-            200,
-            'http://viewer-abcdefg-service.test-ns.svc.cluster.local:80/tensorboard/viewer-abcdefg/',
-            err => {
-              expect(k8sGetCustomObjectSpy.mock.calls[0]).toMatchInlineSnapshot(`
-                Array [
-                  "kubeflow.org",
-                  "v1beta1",
-                  "test-ns",
-                  "viewers",
-                  "viewer-5e1404e679e27b0f0b8ecee8fe515830eaa736c5",
-                ]
-              `);
-              expect(k8sCreateCustomObjectSpy.mock.calls[0]).toMatchInlineSnapshot(`
-                Array [
-                  "kubeflow.org",
-                  "v1beta1",
-                  "test-ns",
-                  "viewers",
-                  Object {
-                    "apiVersion": "kubeflow.org/v1beta1",
-                    "kind": "Viewer",
-                    "metadata": Object {
-                      "name": "viewer-5e1404e679e27b0f0b8ecee8fe515830eaa736c5",
-                      "namespace": "test-ns",
-                    },
-                    "spec": Object {
-                      "podTemplateSpec": Object {
-                        "spec": Object {
-                          "containers": Array [
-                            Object {},
-                          ],
-                        },
-                      },
-                      "tensorboardSpec": Object {
-                        "logDir": "log-dir-1",
-                        "tensorflowImage": "tensorflow/tensorflow:2.0.0",
-                      },
-                      "type": "tensorboard",
-                    },
-                  },
-                ]
-              `);
-              expect(k8sGetCustomObjectSpy.mock.calls[1]).toMatchInlineSnapshot(`
-                Array [
-                  "kubeflow.org",
-                  "v1beta1",
-                  "test-ns",
-                  "viewers",
-                  "viewer-5e1404e679e27b0f0b8ecee8fe515830eaa736c5",
-                ]
-              `);
-              done(err);
-            },
-          );
-      });
-
-      it('creates tensorboard viewer with exist volume', done => {
-        let getRequestCount = 0;
-        k8sGetCustomObjectSpy.mockImplementation(() => {
-          ++getRequestCount;
-          switch (getRequestCount) {
-            case 1:
-              return Promise.reject('Not found');
-            case 2:
-              return Promise.resolve(
-                newGetTensorboardResponse({
-                  name: 'viewer-abcdefg',
-                  logDir: 'Series1:/logs/log-dir-1,Series2:/logs/log-dir-2',
-                  tensorflowImage: 'tensorflow:2.0.0',
-                }),
-              );
-            default:
-              throw new Error('only expected to be called twice in this test');
-          }
-        });
-        k8sCreateCustomObjectSpy.mockImplementation(() => Promise.resolve());
-
-        const tempPath = path.join(fs.mkdtempSync(os.tmpdir()), 'config.json');
-        fs.writeFileSync(tempPath, JSON.stringify(POD_TEMPLATE_SPEC));
-        app = new UIServer(
-          loadConfigs(argv, { VIEWER_TENSORBOARD_POD_TEMPLATE_SPEC_PATH: tempPath }),
-        );
-
-        requests(app.start())
-          .post(
-            `/apps/tensorboard?logdir=${encodeURIComponent(
-              'Series1:volume://tensorboard/log-dir-1,Series2:volume://tensorboard/log-dir-2',
-            )}&namespace=test-ns&tfversion=2.0.0`,
-          )
-          .expect(
-            200,
-            'http://viewer-abcdefg-service.test-ns.svc.cluster.local:80/tensorboard/viewer-abcdefg/',
-            err => {
-              expect(k8sGetCustomObjectSpy.mock.calls[0]).toMatchInlineSnapshot(`
-                Array [
-                  "kubeflow.org",
-                  "v1beta1",
-                  "test-ns",
-                  "viewers",
-                  "viewer-a800f945f0934d978f9cce9959b82ff44dac8493",
-                ]
-              `);
-              expect(k8sCreateCustomObjectSpy.mock.calls[0]).toMatchInlineSnapshot(`
-                Array [
-                  "kubeflow.org",
-                  "v1beta1",
-                  "test-ns",
-                  "viewers",
-                  Object {
-                    "apiVersion": "kubeflow.org/v1beta1",
-                    "kind": "Viewer",
-                    "metadata": Object {
-                      "name": "viewer-a800f945f0934d978f9cce9959b82ff44dac8493",
-                      "namespace": "test-ns",
-                    },
-                    "spec": Object {
-                      "podTemplateSpec": Object {
-                        "spec": Object {
-                          "containers": Array [
-                            Object {
-                              "volumeMounts": Array [
-                                Object {
-                                  "mountPath": "/logs",
-                                  "name": "tensorboard",
-                                },
-                                Object {
-                                  "mountPath": "/data",
-                                  "name": "data",
-                                  "subPath": "tensorboard",
-                                },
-                              ],
-                            },
-                          ],
-                          "volumes": Array [
-                            Object {
-                              "name": "tensorboard",
-                              "persistentVolumeClaim": Object {
-                                "claimName": "logs",
-                              },
-                            },
-                            Object {
-                              "name": "data",
-                              "persistentVolumeClaim": Object {
-                                "claimName": "data",
-                              },
-                            },
-                          ],
-                        },
-                      },
-                      "tensorboardSpec": Object {
-                        "logDir": "Series1:/logs/log-dir-1,Series2:/logs/log-dir-2",
-                        "tensorflowImage": "tensorflow/tensorflow:2.0.0",
-                      },
-                      "type": "tensorboard",
-                    },
-                  },
-                ]
-              `);
-              expect(k8sGetCustomObjectSpy.mock.calls[1]).toMatchInlineSnapshot(`
-                Array [
-                  "kubeflow.org",
-                  "v1beta1",
-                  "test-ns",
-                  "viewers",
-                  "viewer-a800f945f0934d978f9cce9959b82ff44dac8493",
-                ]
-              `);
-              done(err);
-            },
-          );
-      });
-
-      it('creates tensorboard viewer with exist subPath volume', done => {
-        let getRequestCount = 0;
-        k8sGetCustomObjectSpy.mockImplementation(() => {
-          ++getRequestCount;
-          switch (getRequestCount) {
-            case 1:
-              return Promise.reject('Not found');
-            case 2:
-              return Promise.resolve(
-                newGetTensorboardResponse({
-                  name: 'viewer-abcdefg',
-                  logDir: 'Series1:/data/log-dir-1,Series2:/data/log-dir-2',
-                  tensorflowImage: 'tensorflow:2.0.0',
-                }),
-              );
-            default:
-              throw new Error('only expected to be called twice in this test');
-          }
-        });
-        k8sCreateCustomObjectSpy.mockImplementation(() => Promise.resolve());
-
-        const tempPath = path.join(fs.mkdtempSync(os.tmpdir()), 'config.json');
-        fs.writeFileSync(tempPath, JSON.stringify(POD_TEMPLATE_SPEC));
-        app = new UIServer(
-          loadConfigs(argv, { VIEWER_TENSORBOARD_POD_TEMPLATE_SPEC_PATH: tempPath }),
-        );
-
-        requests(app.start())
-          .post(
-            `/apps/tensorboard?logdir=${encodeURIComponent(
-              'Series1:volume://data/tensorboard/log-dir-1,Series2:volume://data/tensorboard/log-dir-2',
-            )}&namespace=test-ns&tfversion=2.0.0`,
-          )
-          .expect(
-            200,
-            'http://viewer-abcdefg-service.test-ns.svc.cluster.local:80/tensorboard/viewer-abcdefg/',
-            err => {
-              expect(k8sGetCustomObjectSpy.mock.calls[0]).toMatchInlineSnapshot(`
-                Array [
-                  "kubeflow.org",
-                  "v1beta1",
-                  "test-ns",
-                  "viewers",
-                  "viewer-82d7d06a6ecb1e4dcba66d06b884d6445a88e4ca",
-                ]
-              `);
-              expect(k8sCreateCustomObjectSpy.mock.calls[0]).toMatchInlineSnapshot(`
-                Array [
-                  "kubeflow.org",
-                  "v1beta1",
-                  "test-ns",
-                  "viewers",
-                  Object {
-                    "apiVersion": "kubeflow.org/v1beta1",
-                    "kind": "Viewer",
-                    "metadata": Object {
-                      "name": "viewer-82d7d06a6ecb1e4dcba66d06b884d6445a88e4ca",
-                      "namespace": "test-ns",
-                    },
-                    "spec": Object {
-                      "podTemplateSpec": Object {
-                        "spec": Object {
-                          "containers": Array [
-                            Object {
-                              "volumeMounts": Array [
-                                Object {
-                                  "mountPath": "/logs",
-                                  "name": "tensorboard",
-                                },
-                                Object {
-                                  "mountPath": "/data",
-                                  "name": "data",
-                                  "subPath": "tensorboard",
-                                },
-                              ],
-                            },
-                          ],
-                          "volumes": Array [
-                            Object {
-                              "name": "tensorboard",
-                              "persistentVolumeClaim": Object {
-                                "claimName": "logs",
-                              },
-                            },
-                            Object {
-                              "name": "data",
-                              "persistentVolumeClaim": Object {
-                                "claimName": "data",
-                              },
-                            },
-                          ],
-                        },
-                      },
-                      "tensorboardSpec": Object {
-                        "logDir": "Series1:/data/log-dir-1,Series2:/data/log-dir-2",
-                        "tensorflowImage": "tensorflow/tensorflow:2.0.0",
-                      },
-                      "type": "tensorboard",
-                    },
-                  },
-                ]
-              `);
-              expect(k8sGetCustomObjectSpy.mock.calls[1]).toMatchInlineSnapshot(`
-                Array [
-                  "kubeflow.org",
-                  "v1beta1",
-                  "test-ns",
-                  "viewers",
-                  "viewer-82d7d06a6ecb1e4dcba66d06b884d6445a88e4ca",
-                ]
-              `);
-              done(err);
-            },
-          );
-      });
-
-      it('creates tensorboard viewer with not exist volume and return error', done => {
-        const errorSpy = jest.spyOn(console, 'error');
-        errorSpy.mockImplementation();
-
-        k8sGetCustomObjectSpy.mockImplementation(() => {
-          return Promise.reject('Not found');
-        });
-
-        const tempPath = path.join(fs.mkdtempSync(os.tmpdir()), 'config.json');
-        fs.writeFileSync(tempPath, JSON.stringify(POD_TEMPLATE_SPEC));
-        app = new UIServer(
-          loadConfigs(argv, { VIEWER_TENSORBOARD_POD_TEMPLATE_SPEC_PATH: tempPath }),
-        );
-
-        requests(app.start())
-          .post(
-            `/apps/tensorboard?logdir=${encodeURIComponent(
-              'volume://notexistvolume/logs/log-dir-1',
-            )}&namespace=test-ns&tfversion=2.0.0`,
-          )
-          .expect(
-            500,
-            `Failed to start Tensorboard app: Cannot find file "volume://notexistvolume/logs/log-dir-1" in pod "unknown": volume "notexistvolume" not configured`,
-            err => {
-              expect(errorSpy).toHaveBeenCalledTimes(1);
-              done(err);
-            },
-          );
-      });
-
-      it('creates tensorboard viewer with not exist subPath volume mount and return error', done => {
-        const errorSpy = jest.spyOn(console, 'error');
-        errorSpy.mockImplementation();
-
-        k8sGetCustomObjectSpy.mockImplementation(() => {
-          return Promise.reject('Not found');
-        });
-
-        const tempPath = path.join(fs.mkdtempSync(os.tmpdir()), 'config.json');
-        fs.writeFileSync(tempPath, JSON.stringify(POD_TEMPLATE_SPEC));
-        app = new UIServer(
-          loadConfigs(argv, { VIEWER_TENSORBOARD_POD_TEMPLATE_SPEC_PATH: tempPath }),
-        );
-
-        requests(app.start())
-          .post(
-            `/apps/tensorboard?logdir=${encodeURIComponent(
-              'volume://data/notexit/mountnotexist/log-dir-1',
-            )}&namespace=test-ns&tfversion=2.0.0`,
-          )
-          .expect(
-            500,
-            `Failed to start Tensorboard app: Cannot find file "volume://data/notexit/mountnotexist/log-dir-1" in pod "unknown": volume "data" not mounted or volume "data" with subPath (which is prefix of notexit/mountnotexist/log-dir-1) not mounted`,
-            err => {
-              expect(errorSpy).toHaveBeenCalledTimes(1);
-              done(err);
-            },
-          );
-      });
-
-      it('returns error when there is an existing tensorboard with different version', done => {
-        const errorSpy = jest.spyOn(console, 'error');
-        errorSpy.mockImplementation();
-        k8sGetCustomObjectSpy.mockImplementation(() =>
-          Promise.resolve(
-            newGetTensorboardResponse({
-              name: 'viewer-abcdefg',
-              logDir: 'log-dir-1',
-              tensorflowImage: 'tensorflow:2.1.0',
-            }),
-          ),
-        );
-        k8sCreateCustomObjectSpy.mockImplementation(() => Promise.resolve());
-
-        app = new UIServer(loadConfigs(argv, {}));
-        requests(app.start())
-          .post(
-            `/apps/tensorboard?logdir=${encodeURIComponent(
-              'log-dir-1',
-            )}&namespace=test-ns&tfversion=2.0.0`,
-          )
-          .expect(
-            500,
-            `Failed to start Tensorboard app: There's already an existing tensorboard instance with a different version 2.1.0`,
-            err => {
-              expect(errorSpy).toHaveBeenCalledTimes(1);
-              done(err);
-            },
-          );
-      });
-
-      it('returns existing pod address if there is an existing tensorboard with the same version', done => {
-        k8sGetCustomObjectSpy.mockImplementation(() =>
-          Promise.resolve(
-            newGetTensorboardResponse({
-              name: 'viewer-abcdefg',
-              logDir: 'log-dir-1',
-              tensorflowImage: 'tensorflow:2.0.0',
-            }),
-          ),
-        );
-        k8sCreateCustomObjectSpy.mockImplementation(() => Promise.resolve());
-
-        app = new UIServer(loadConfigs(argv, {}));
-        requests(app.start())
-          .post(
-            `/apps/tensorboard?logdir=${encodeURIComponent(
-              'log-dir-1',
-            )}&namespace=test-ns&tfversion=2.0.0`,
-          )
-          .expect(
-            200,
-            'http://viewer-abcdefg-service.test-ns.svc.cluster.local:80/tensorboard/viewer-abcdefg/',
-            done,
-          );
-      });
-    });
-
-    describe('delete', () => {
-      it('requires logdir', done => {
-        app = new UIServer(loadConfigs(argv, {}));
-        requests(app.start())
-          .delete('/apps/tensorboard')
-          .expect(400, 'logdir argument is required', done);
-      });
-
-      it('requires namespace', done => {
-        app = new UIServer(loadConfigs(argv, {}));
-        requests(app.start())
-          .delete('/apps/tensorboard?logdir=some-log-dir')
-          .expect(400, 'namespace argument is required', done);
-      });
-
-      it('deletes tensorboard viewer custom object', done => {
-        k8sGetCustomObjectSpy.mockImplementation(() =>
-          Promise.resolve(
-            newGetTensorboardResponse({
-              name: 'viewer-abcdefg',
-              logDir: 'log-dir-1',
-              tensorflowImage: 'tensorflow:2.0.0',
-            }),
-          ),
-        );
-        k8sDeleteCustomObjectSpy.mockImplementation(() => Promise.resolve());
-
-        app = new UIServer(loadConfigs(argv, {}));
-        requests(app.start())
-          .delete(`/apps/tensorboard?logdir=${encodeURIComponent('log-dir-1')}&namespace=test-ns`)
-          .expect(200, 'Tensorboard deleted.', err => {
-            expect(k8sDeleteCustomObjectSpy.mock.calls[0]).toMatchInlineSnapshot(`
-              Array [
-                "kubeflow.org",
-                "v1beta1",
-                "test-ns",
-                "viewers",
-                "viewer-5e1404e679e27b0f0b8ecee8fe515830eaa736c5",
-                V1DeleteOptions {},
-              ]
-            `);
-            done(err);
-          });
-      });
-    });
-  });
-
   // TODO: Add integration tests for k8s helper related endpoints
   // describe('/k8s/pod/logs', () => {});
 
diff --git a/frontend/server/integration-tests/artifact-get.test.ts b/frontend/server/integration-tests/artifact-get.test.ts
--- a/frontend/server/integration-tests/artifact-get.test.ts
+++ b/frontend/server/integration-tests/artifact-get.test.ts
@@ -16,7 +16,6 @@ import { Storage as GCSStorage } from '@google-cloud/storage';
 import * as fs from 'fs';
 import * as minio from 'minio';
 import fetch from 'node-fetch';
-import * as os from 'os';
 import * as path from 'path';
 import { PassThrough } from 'stream';
 import requests from 'supertest';
@@ -24,7 +23,7 @@ import { UIServer } from '../app';
 import { loadConfigs } from '../configs';
 import * as serverInfo from '../helpers/server-info';
 import * as minioHelper from '../minio-helper';
-import { commonSetup } from './test-helper';
+import { commonSetup, mkTempDir } from './test-helper';
 
 const MinioClient = minio.Client;
 jest.mock('minio');
@@ -290,7 +289,7 @@ describe('/artifacts', () => {
 
     it('responds with a volume artifact if source=volume', done => {
       const artifactContent = 'hello world';
-      const tempPath = path.join(fs.mkdtempSync(os.tmpdir()), 'content');
+      const tempPath = path.join(mkTempDir(), 'content');
       fs.writeFileSync(tempPath, artifactContent);
 
       jest.spyOn(serverInfo, 'getHostPod').mockImplementation(() =>
@@ -334,7 +333,7 @@ describe('/artifacts', () => {
 
     it('responds with a partial volume artifact if peek=5 is set', done => {
       const artifactContent = 'hello world';
-      const tempPath = path.join(fs.mkdtempSync(os.tmpdir()), 'content');
+      const tempPath = path.join(mkTempDir(), 'content');
       fs.writeFileSync(tempPath, artifactContent);
 
       jest.spyOn(serverInfo, 'getHostPod').mockImplementation(() =>
diff --git a/frontend/server/integration-tests/tensorboard.test.ts b/frontend/server/integration-tests/tensorboard.test.ts
new file mode 100644
--- /dev/null
+++ b/frontend/server/integration-tests/tensorboard.test.ts
@@ -0,0 +1,829 @@
+// Copyright 2021 The Kubeflow Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+import express from 'express';
+import * as fs from 'fs';
+import { Server } from 'http';
+import * as path from 'path';
+import requests from 'supertest';
+import { UIServer } from '../app';
+import { loadConfigs } from '../configs';
+import { TEST_ONLY as K8S_TEST_EXPORT } from '../k8s-helper';
+import { buildQuery, commonSetup, mkTempDir } from './test-helper';
+
+beforeEach(() => {
+  jest.spyOn(global.console, 'info').mockImplementation();
+  jest.spyOn(global.console, 'log').mockImplementation();
+  jest.spyOn(global.console, 'debug').mockImplementation();
+});
+
+describe('/apps/tensorboard', () => {
+  let app: UIServer;
+  afterEach(() => {
+    if (app) {
+      app.close();
+    }
+  });
+  const tagName = '1.0.0';
+  const commitHash = 'abcdefg';
+  const { argv } = commonSetup({ tagName, commitHash });
+
+  const POD_TEMPLATE_SPEC = {
+    spec: {
+      containers: [
+        {
+          volumeMounts: [
+            {
+              name: 'tensorboard',
+              mountPath: '/logs',
+            },
+            {
+              name: 'data',
+              subPath: 'tensorboard',
+              mountPath: '/data',
+            },
+          ],
+        },
+      ],
+      volumes: [
+        {
+          name: 'tensorboard',
+          persistentVolumeClaim: {
+            claimName: 'logs',
+          },
+        },
+        {
+          name: 'data',
+          persistentVolumeClaim: {
+            claimName: 'data',
+          },
+        },
+      ],
+    },
+  };
+
+  let k8sGetCustomObjectSpy: jest.SpyInstance;
+  let k8sDeleteCustomObjectSpy: jest.SpyInstance;
+  let k8sCreateCustomObjectSpy: jest.SpyInstance;
+  let kfpApiServer: Server;
+
+  function newGetTensorboardResponse({
+    name = 'viewer-example',
+    logDir = 'log-dir-example',
+    tensorflowImage = 'tensorflow:2.0.0',
+    type = 'tensorboard',
+  }: {
+    name?: string;
+    logDir?: string;
+    tensorflowImage?: string;
+    type?: string;
+  } = {}) {
+    return {
+      response: undefined as any, // unused
+      body: {
+        metadata: {
+          name,
+        },
+        spec: {
+          tensorboardSpec: { logDir, tensorflowImage },
+          type,
+        },
+      },
+    };
+  }
+
+  beforeEach(() => {
+    k8sGetCustomObjectSpy = jest.spyOn(
+      K8S_TEST_EXPORT.k8sV1CustomObjectClient,
+      'getNamespacedCustomObject',
+    );
+    k8sDeleteCustomObjectSpy = jest.spyOn(
+      K8S_TEST_EXPORT.k8sV1CustomObjectClient,
+      'deleteNamespacedCustomObject',
+    );
+    k8sCreateCustomObjectSpy = jest.spyOn(
+      K8S_TEST_EXPORT.k8sV1CustomObjectClient,
+      'createNamespacedCustomObject',
+    );
+  });
+
+  afterEach(() => {
+    if (kfpApiServer) {
+      kfpApiServer.close();
+    }
+  });
+
+  describe('get', () => {
+    it('requires logdir for get tensorboard', done => {
+      app = new UIServer(loadConfigs(argv, {}));
+      requests(app.start())
+        .get('/apps/tensorboard')
+        .expect(400, 'logdir argument is required', done);
+    });
+
+    it('requires namespace for get tensorboard', done => {
+      app = new UIServer(loadConfigs(argv, {}));
+      requests(app.start())
+        .get('/apps/tensorboard?logdir=some-log-dir')
+        .expect(400, 'namespace argument is required', done);
+    });
+
+    it('does not crash with a weird query', done => {
+      app = new UIServer(loadConfigs(argv, {}));
+      k8sGetCustomObjectSpy.mockImplementation(() => Promise.resolve(newGetTensorboardResponse()));
+      // The special case is that, decodeURIComponent('%2') throws an
+      // exception, so this can verify handler doesn't do extra
+      // decodeURIComponent on queries.
+      const weirdLogDir = encodeURIComponent('%2');
+      requests(app.start())
+        .get(`/apps/tensorboard?logdir=${weirdLogDir}&namespace=test-ns`)
+        .expect(200, done);
+    });
+
+    function setupMockKfpApiService({ port = 3001 }: { port?: number } = {}) {
+      const receivedHeaders: any[] = [];
+      kfpApiServer = express()
+        .get('/apis/v1beta1/auth', (req, res) => {
+          receivedHeaders.push(req.headers);
+          res.status(200).send('{}'); // Authorized
+        })
+        .listen(port);
+      return { receivedHeaders, host: 'localhost', port };
+    }
+
+    it('authorizes user requests from KFP auth api', done => {
+      const { receivedHeaders, host, port } = setupMockKfpApiService();
+      app = new UIServer(
+        loadConfigs(argv, {
+          ENABLE_AUTHZ: 'true',
+          ML_PIPELINE_SERVICE_PORT: `${port}`,
+          ML_PIPELINE_SERVICE_HOST: host,
+        }),
+      );
+      k8sGetCustomObjectSpy.mockImplementation(() => Promise.resolve(newGetTensorboardResponse()));
+      requests(app.start())
+        .get(`/apps/tensorboard?logdir=some-log-dir&namespace=test-ns`)
+        .set('x-goog-authenticated-user-email', 'accounts.google.com:user@google.com')
+        .expect(200, err => {
+          expect(receivedHeaders).toHaveLength(1);
+          expect(receivedHeaders[0]).toMatchInlineSnapshot(`
+              Object {
+                "accept": "*/*",
+                "accept-encoding": "gzip,deflate",
+                "connection": "close",
+                "host": "localhost:3001",
+                "user-agent": "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)",
+                "x-goog-authenticated-user-email": "accounts.google.com:user@google.com",
+              }
+            `);
+          done(err);
+        });
+    });
+
+    it('uses configured KUBEFLOW_USERID_HEADER for user identity', done => {
+      const { receivedHeaders, host, port } = setupMockKfpApiService();
+      app = new UIServer(
+        loadConfigs(argv, {
+          ENABLE_AUTHZ: 'true',
+          KUBEFLOW_USERID_HEADER: 'x-kubeflow-userid',
+          ML_PIPELINE_SERVICE_PORT: `${port}`,
+          ML_PIPELINE_SERVICE_HOST: host,
+        }),
+      );
+      k8sGetCustomObjectSpy.mockImplementation(() => Promise.resolve(newGetTensorboardResponse()));
+      requests(app.start())
+        .get(`/apps/tensorboard?logdir=some-log-dir&namespace=test-ns`)
+        .set('x-kubeflow-userid', 'user@kubeflow.org')
+        .expect(200, err => {
+          expect(receivedHeaders).toHaveLength(1);
+          expect(receivedHeaders[0]).toHaveProperty('x-kubeflow-userid', 'user@kubeflow.org');
+          done(err);
+        });
+    });
+
+    it('rejects user requests when KFP auth api rejected', done => {
+      const errorSpy = jest.spyOn(console, 'error');
+      errorSpy.mockImplementation();
+
+      const apiServerPort = 3001;
+      kfpApiServer = express()
+        .get('/apis/v1beta1/auth', (_, res) => {
+          res.status(400).send(
+            JSON.stringify({
+              error: 'User xxx is not unauthorized to list viewers',
+              details: ['unauthorized', 'callstack'],
+            }),
+          );
+        })
+        .listen(apiServerPort);
+      app = new UIServer(
+        loadConfigs(argv, {
+          ENABLE_AUTHZ: 'true',
+          ML_PIPELINE_SERVICE_PORT: `${apiServerPort}`,
+          ML_PIPELINE_SERVICE_HOST: 'localhost',
+        }),
+      );
+      k8sGetCustomObjectSpy.mockImplementation(() => Promise.resolve(newGetTensorboardResponse()));
+      requests(app.start())
+        .get(`/apps/tensorboard?logdir=some-log-dir&namespace=test-ns`)
+        .expect(
+          401,
+          'User is not authorized to GET VIEWERS in namespace test-ns: User xxx is not unauthorized to list viewers',
+          err => {
+            expect(errorSpy).toHaveBeenCalledTimes(1);
+            expect(
+              errorSpy,
+            ).toHaveBeenCalledWith(
+              'User is not authorized to GET VIEWERS in namespace test-ns: User xxx is not unauthorized to list viewers',
+              ['unauthorized', 'callstack'],
+            );
+            done(err);
+          },
+        );
+    });
+
+    it('gets tensorboard url, version and image', done => {
+      app = new UIServer(loadConfigs(argv, {}));
+      k8sGetCustomObjectSpy.mockImplementation(() =>
+        Promise.resolve(
+          newGetTensorboardResponse({
+            name: 'viewer-abcdefg',
+            logDir: 'log-dir-1',
+            tensorflowImage: 'tensorflow:2.0.0',
+          }),
+        ),
+      );
+
+      requests(app.start())
+        .get(`/apps/tensorboard?logdir=${encodeURIComponent('log-dir-1')}&namespace=test-ns`)
+        .expect(
+          200,
+          JSON.stringify({
+            podAddress:
+              'http://viewer-abcdefg-service.test-ns.svc.cluster.local:80/tensorboard/viewer-abcdefg/',
+            tfVersion: '2.0.0',
+            image: 'tensorflow:2.0.0',
+          }),
+          err => {
+            expect(k8sGetCustomObjectSpy.mock.calls[0]).toMatchInlineSnapshot(`
+                              Array [
+                                "kubeflow.org",
+                                "v1beta1",
+                                "test-ns",
+                                "viewers",
+                                "viewer-5e1404e679e27b0f0b8ecee8fe515830eaa736c5",
+                              ]
+                          `);
+            done(err);
+          },
+        );
+    });
+  });
+
+  describe('post (create)', () => {
+    it('requires logdir', done => {
+      app = new UIServer(loadConfigs(argv, {}));
+      requests(app.start())
+        .post('/apps/tensorboard')
+        .expect(400, 'logdir argument is required', done);
+    });
+
+    it('requires namespace', done => {
+      app = new UIServer(loadConfigs(argv, {}));
+      requests(app.start())
+        .post('/apps/tensorboard?logdir=some-log-dir')
+        .expect(400, 'namespace argument is required', done);
+    });
+
+    it('requires tfversion or image', done => {
+      app = new UIServer(loadConfigs(argv, {}));
+      requests(app.start())
+        .post('/apps/tensorboard?logdir=some-log-dir&namespace=test-ns')
+        .expect(400, 'missing required argument: tfversion (tensorflow version) or image', done);
+    });
+
+    it('creates tensorboard viewer custom object and waits for it', done => {
+      let getRequestCount = 0;
+      k8sGetCustomObjectSpy.mockImplementation(() => {
+        ++getRequestCount;
+        switch (getRequestCount) {
+          case 1:
+            return Promise.reject('Not found');
+          case 2:
+            return Promise.resolve(
+              newGetTensorboardResponse({
+                name: 'viewer-abcdefg',
+                logDir: 'log-dir-1',
+                tensorflowImage: 'tensorflow:2.0.0',
+              }),
+            );
+          default:
+            throw new Error('only expected to be called twice in this test');
+        }
+      });
+      k8sCreateCustomObjectSpy.mockImplementation(() => Promise.resolve());
+
+      app = new UIServer(loadConfigs(argv, {}));
+      requests(app.start())
+        .post(
+          `/apps/tensorboard?logdir=${encodeURIComponent(
+            'log-dir-1',
+          )}&namespace=test-ns&tfversion=2.0.0`,
+        )
+        .expect(
+          200,
+          'http://viewer-abcdefg-service.test-ns.svc.cluster.local:80/tensorboard/viewer-abcdefg/',
+          err => {
+            expect(k8sGetCustomObjectSpy.mock.calls[0]).toMatchInlineSnapshot(`
+                Array [
+                  "kubeflow.org",
+                  "v1beta1",
+                  "test-ns",
+                  "viewers",
+                  "viewer-5e1404e679e27b0f0b8ecee8fe515830eaa736c5",
+                ]
+              `);
+            expect(k8sCreateCustomObjectSpy.mock.calls[0]).toMatchInlineSnapshot(`
+                Array [
+                  "kubeflow.org",
+                  "v1beta1",
+                  "test-ns",
+                  "viewers",
+                  Object {
+                    "apiVersion": "kubeflow.org/v1beta1",
+                    "kind": "Viewer",
+                    "metadata": Object {
+                      "name": "viewer-5e1404e679e27b0f0b8ecee8fe515830eaa736c5",
+                      "namespace": "test-ns",
+                    },
+                    "spec": Object {
+                      "podTemplateSpec": Object {
+                        "spec": Object {
+                          "containers": Array [
+                            Object {},
+                          ],
+                        },
+                      },
+                      "tensorboardSpec": Object {
+                        "logDir": "log-dir-1",
+                        "tensorflowImage": "tensorflow/tensorflow:2.0.0",
+                      },
+                      "type": "tensorboard",
+                    },
+                  },
+                ]
+              `);
+            expect(k8sGetCustomObjectSpy.mock.calls[1]).toMatchInlineSnapshot(`
+                Array [
+                  "kubeflow.org",
+                  "v1beta1",
+                  "test-ns",
+                  "viewers",
+                  "viewer-5e1404e679e27b0f0b8ecee8fe515830eaa736c5",
+                ]
+              `);
+            done(err);
+          },
+        );
+    });
+
+    it('creates tensorboard viewer with specified image', done => {
+      let getRequestCount = 0;
+      const image = 'gcr.io/deeplearning-platform-release/tf2-cpu.2-4';
+      k8sGetCustomObjectSpy.mockImplementation(() => {
+        ++getRequestCount;
+        switch (getRequestCount) {
+          case 1:
+            return Promise.reject('Not found');
+          case 2:
+            return Promise.resolve(
+              newGetTensorboardResponse({
+                tensorflowImage: image,
+              }),
+            );
+          default:
+            throw new Error('only expected to be called twice in this test');
+        }
+      });
+      k8sCreateCustomObjectSpy.mockImplementation(() => Promise.resolve());
+
+      app = new UIServer(loadConfigs(argv, {}));
+      requests(app.start())
+        .post(
+          `/apps/tensorboard${buildQuery({ logdir: 'log-dir-1', namespace: 'test-ns', image })}`,
+        )
+        .expect(200, err => {
+          expect(
+            k8sCreateCustomObjectSpy.mock.calls[0][4].spec.tensorboardSpec.tensorflowImage,
+          ).toEqual(image);
+          done(err);
+        });
+    });
+
+    it('creates tensorboard viewer with exist volume', done => {
+      let getRequestCount = 0;
+      k8sGetCustomObjectSpy.mockImplementation(() => {
+        ++getRequestCount;
+        switch (getRequestCount) {
+          case 1:
+            return Promise.reject('Not found');
+          case 2:
+            return Promise.resolve(
+              newGetTensorboardResponse({
+                name: 'viewer-abcdefg',
+                logDir: 'Series1:/logs/log-dir-1,Series2:/logs/log-dir-2',
+                tensorflowImage: 'tensorflow:2.0.0',
+              }),
+            );
+          default:
+            throw new Error('only expected to be called twice in this test');
+        }
+      });
+      k8sCreateCustomObjectSpy.mockImplementation(() => Promise.resolve());
+
+      const tempPath = path.join(mkTempDir(), 'config.json');
+      fs.writeFileSync(tempPath, JSON.stringify(POD_TEMPLATE_SPEC));
+      app = new UIServer(
+        loadConfigs(argv, { VIEWER_TENSORBOARD_POD_TEMPLATE_SPEC_PATH: tempPath }),
+      );
+
+      requests(app.start())
+        .post(
+          `/apps/tensorboard?logdir=${encodeURIComponent(
+            'Series1:volume://tensorboard/log-dir-1,Series2:volume://tensorboard/log-dir-2',
+          )}&namespace=test-ns&tfversion=2.0.0`,
+        )
+        .expect(
+          200,
+          'http://viewer-abcdefg-service.test-ns.svc.cluster.local:80/tensorboard/viewer-abcdefg/',
+          err => {
+            expect(k8sGetCustomObjectSpy.mock.calls[0]).toMatchInlineSnapshot(`
+                Array [
+                  "kubeflow.org",
+                  "v1beta1",
+                  "test-ns",
+                  "viewers",
+                  "viewer-a800f945f0934d978f9cce9959b82ff44dac8493",
+                ]
+              `);
+            expect(k8sCreateCustomObjectSpy.mock.calls[0]).toMatchInlineSnapshot(`
+                Array [
+                  "kubeflow.org",
+                  "v1beta1",
+                  "test-ns",
+                  "viewers",
+                  Object {
+                    "apiVersion": "kubeflow.org/v1beta1",
+                    "kind": "Viewer",
+                    "metadata": Object {
+                      "name": "viewer-a800f945f0934d978f9cce9959b82ff44dac8493",
+                      "namespace": "test-ns",
+                    },
+                    "spec": Object {
+                      "podTemplateSpec": Object {
+                        "spec": Object {
+                          "containers": Array [
+                            Object {
+                              "volumeMounts": Array [
+                                Object {
+                                  "mountPath": "/logs",
+                                  "name": "tensorboard",
+                                },
+                                Object {
+                                  "mountPath": "/data",
+                                  "name": "data",
+                                  "subPath": "tensorboard",
+                                },
+                              ],
+                            },
+                          ],
+                          "volumes": Array [
+                            Object {
+                              "name": "tensorboard",
+                              "persistentVolumeClaim": Object {
+                                "claimName": "logs",
+                              },
+                            },
+                            Object {
+                              "name": "data",
+                              "persistentVolumeClaim": Object {
+                                "claimName": "data",
+                              },
+                            },
+                          ],
+                        },
+                      },
+                      "tensorboardSpec": Object {
+                        "logDir": "Series1:/logs/log-dir-1,Series2:/logs/log-dir-2",
+                        "tensorflowImage": "tensorflow/tensorflow:2.0.0",
+                      },
+                      "type": "tensorboard",
+                    },
+                  },
+                ]
+              `);
+            expect(k8sGetCustomObjectSpy.mock.calls[1]).toMatchInlineSnapshot(`
+                Array [
+                  "kubeflow.org",
+                  "v1beta1",
+                  "test-ns",
+                  "viewers",
+                  "viewer-a800f945f0934d978f9cce9959b82ff44dac8493",
+                ]
+              `);
+            done(err);
+          },
+        );
+    });
+
+    it('creates tensorboard viewer with exist subPath volume', done => {
+      let getRequestCount = 0;
+      k8sGetCustomObjectSpy.mockImplementation(() => {
+        ++getRequestCount;
+        switch (getRequestCount) {
+          case 1:
+            return Promise.reject('Not found');
+          case 2:
+            return Promise.resolve(
+              newGetTensorboardResponse({
+                name: 'viewer-abcdefg',
+                logDir: 'Series1:/data/log-dir-1,Series2:/data/log-dir-2',
+                tensorflowImage: 'tensorflow:2.0.0',
+              }),
+            );
+          default:
+            throw new Error('only expected to be called twice in this test');
+        }
+      });
+      k8sCreateCustomObjectSpy.mockImplementation(() => Promise.resolve());
+
+      const tempPath = path.join(mkTempDir(), 'config.json');
+      fs.writeFileSync(tempPath, JSON.stringify(POD_TEMPLATE_SPEC));
+      app = new UIServer(
+        loadConfigs(argv, { VIEWER_TENSORBOARD_POD_TEMPLATE_SPEC_PATH: tempPath }),
+      );
+
+      requests(app.start())
+        .post(
+          `/apps/tensorboard?logdir=${encodeURIComponent(
+            'Series1:volume://data/tensorboard/log-dir-1,Series2:volume://data/tensorboard/log-dir-2',
+          )}&namespace=test-ns&tfversion=2.0.0`,
+        )
+        .expect(
+          200,
+          'http://viewer-abcdefg-service.test-ns.svc.cluster.local:80/tensorboard/viewer-abcdefg/',
+          err => {
+            expect(k8sGetCustomObjectSpy.mock.calls[0]).toMatchInlineSnapshot(`
+                Array [
+                  "kubeflow.org",
+                  "v1beta1",
+                  "test-ns",
+                  "viewers",
+                  "viewer-82d7d06a6ecb1e4dcba66d06b884d6445a88e4ca",
+                ]
+              `);
+            expect(k8sCreateCustomObjectSpy.mock.calls[0]).toMatchInlineSnapshot(`
+                Array [
+                  "kubeflow.org",
+                  "v1beta1",
+                  "test-ns",
+                  "viewers",
+                  Object {
+                    "apiVersion": "kubeflow.org/v1beta1",
+                    "kind": "Viewer",
+                    "metadata": Object {
+                      "name": "viewer-82d7d06a6ecb1e4dcba66d06b884d6445a88e4ca",
+                      "namespace": "test-ns",
+                    },
+                    "spec": Object {
+                      "podTemplateSpec": Object {
+                        "spec": Object {
+                          "containers": Array [
+                            Object {
+                              "volumeMounts": Array [
+                                Object {
+                                  "mountPath": "/logs",
+                                  "name": "tensorboard",
+                                },
+                                Object {
+                                  "mountPath": "/data",
+                                  "name": "data",
+                                  "subPath": "tensorboard",
+                                },
+                              ],
+                            },
+                          ],
+                          "volumes": Array [
+                            Object {
+                              "name": "tensorboard",
+                              "persistentVolumeClaim": Object {
+                                "claimName": "logs",
+                              },
+                            },
+                            Object {
+                              "name": "data",
+                              "persistentVolumeClaim": Object {
+                                "claimName": "data",
+                              },
+                            },
+                          ],
+                        },
+                      },
+                      "tensorboardSpec": Object {
+                        "logDir": "Series1:/data/log-dir-1,Series2:/data/log-dir-2",
+                        "tensorflowImage": "tensorflow/tensorflow:2.0.0",
+                      },
+                      "type": "tensorboard",
+                    },
+                  },
+                ]
+              `);
+            expect(k8sGetCustomObjectSpy.mock.calls[1]).toMatchInlineSnapshot(`
+                Array [
+                  "kubeflow.org",
+                  "v1beta1",
+                  "test-ns",
+                  "viewers",
+                  "viewer-82d7d06a6ecb1e4dcba66d06b884d6445a88e4ca",
+                ]
+              `);
+            done(err);
+          },
+        );
+    });
+
+    it('creates tensorboard viewer with not exist volume and return error', done => {
+      const errorSpy = jest.spyOn(console, 'error');
+      errorSpy.mockImplementation();
+
+      k8sGetCustomObjectSpy.mockImplementation(() => {
+        return Promise.reject('Not found');
+      });
+
+      const tempPath = path.join(mkTempDir(), 'config.json');
+      fs.writeFileSync(tempPath, JSON.stringify(POD_TEMPLATE_SPEC));
+      app = new UIServer(
+        loadConfigs(argv, { VIEWER_TENSORBOARD_POD_TEMPLATE_SPEC_PATH: tempPath }),
+      );
+
+      requests(app.start())
+        .post(
+          `/apps/tensorboard?logdir=${encodeURIComponent(
+            'volume://notexistvolume/logs/log-dir-1',
+          )}&namespace=test-ns&tfversion=2.0.0`,
+        )
+        .expect(
+          500,
+          `Failed to start Tensorboard app: Cannot find file "volume://notexistvolume/logs/log-dir-1" in pod "unknown": volume "notexistvolume" not configured`,
+          err => {
+            expect(errorSpy).toHaveBeenCalledTimes(1);
+            done(err);
+          },
+        );
+    });
+
+    it('creates tensorboard viewer with not exist subPath volume mount and return error', done => {
+      const errorSpy = jest.spyOn(console, 'error');
+      errorSpy.mockImplementation();
+
+      k8sGetCustomObjectSpy.mockImplementation(() => {
+        return Promise.reject('Not found');
+      });
+
+      const tempPath = path.join(mkTempDir(), 'config.json');
+      fs.writeFileSync(tempPath, JSON.stringify(POD_TEMPLATE_SPEC));
+      app = new UIServer(
+        loadConfigs(argv, { VIEWER_TENSORBOARD_POD_TEMPLATE_SPEC_PATH: tempPath }),
+      );
+
+      requests(app.start())
+        .post(
+          `/apps/tensorboard?logdir=${encodeURIComponent(
+            'volume://data/notexit/mountnotexist/log-dir-1',
+          )}&namespace=test-ns&tfversion=2.0.0`,
+        )
+        .expect(
+          500,
+          `Failed to start Tensorboard app: Cannot find file "volume://data/notexit/mountnotexist/log-dir-1" in pod "unknown": volume "data" not mounted or volume "data" with subPath (which is prefix of notexit/mountnotexist/log-dir-1) not mounted`,
+          err => {
+            expect(errorSpy).toHaveBeenCalledTimes(1);
+            done(err);
+          },
+        );
+    });
+
+    it('returns error when there is an existing tensorboard with different version', done => {
+      const errorSpy = jest.spyOn(console, 'error');
+      errorSpy.mockImplementation();
+      k8sGetCustomObjectSpy.mockImplementation(() =>
+        Promise.resolve(
+          newGetTensorboardResponse({
+            name: 'viewer-abcdefg',
+            logDir: 'log-dir-1',
+            tensorflowImage: 'tensorflow:2.1.0',
+          }),
+        ),
+      );
+      k8sCreateCustomObjectSpy.mockImplementation(() => Promise.resolve());
+
+      app = new UIServer(loadConfigs(argv, {}));
+      requests(app.start())
+        .post(
+          `/apps/tensorboard?logdir=${encodeURIComponent(
+            'log-dir-1',
+          )}&namespace=test-ns&tfversion=2.0.0`,
+        )
+        .expect(
+          500,
+          `Failed to start Tensorboard app: There's already an existing tensorboard instance with a different version 2.1.0`,
+          err => {
+            expect(errorSpy).toHaveBeenCalledTimes(1);
+            done(err);
+          },
+        );
+    });
+
+    it('returns existing pod address if there is an existing tensorboard with the same version', done => {
+      k8sGetCustomObjectSpy.mockImplementation(() =>
+        Promise.resolve(
+          newGetTensorboardResponse({
+            name: 'viewer-abcdefg',
+            logDir: 'log-dir-1',
+            tensorflowImage: 'tensorflow:2.0.0',
+          }),
+        ),
+      );
+      k8sCreateCustomObjectSpy.mockImplementation(() => Promise.resolve());
+
+      app = new UIServer(loadConfigs(argv, {}));
+      requests(app.start())
+        .post(
+          `/apps/tensorboard?logdir=${encodeURIComponent(
+            'log-dir-1',
+          )}&namespace=test-ns&tfversion=2.0.0`,
+        )
+        .expect(
+          200,
+          'http://viewer-abcdefg-service.test-ns.svc.cluster.local:80/tensorboard/viewer-abcdefg/',
+          done,
+        );
+    });
+  });
+
+  describe('delete', () => {
+    it('requires logdir', done => {
+      app = new UIServer(loadConfigs(argv, {}));
+      requests(app.start())
+        .delete('/apps/tensorboard')
+        .expect(400, 'logdir argument is required', done);
+    });
+
+    it('requires namespace', done => {
+      app = new UIServer(loadConfigs(argv, {}));
+      requests(app.start())
+        .delete('/apps/tensorboard?logdir=some-log-dir')
+        .expect(400, 'namespace argument is required', done);
+    });
+
+    it('deletes tensorboard viewer custom object', done => {
+      k8sGetCustomObjectSpy.mockImplementation(() =>
+        Promise.resolve(
+          newGetTensorboardResponse({
+            name: 'viewer-abcdefg',
+            logDir: 'log-dir-1',
+            tensorflowImage: 'tensorflow:2.0.0',
+          }),
+        ),
+      );
+      k8sDeleteCustomObjectSpy.mockImplementation(() => Promise.resolve());
+
+      app = new UIServer(loadConfigs(argv, {}));
+      requests(app.start())
+        .delete(`/apps/tensorboard?logdir=${encodeURIComponent('log-dir-1')}&namespace=test-ns`)
+        .expect(200, 'Tensorboard deleted.', err => {
+          expect(k8sDeleteCustomObjectSpy.mock.calls[0]).toMatchInlineSnapshot(`
+              Array [
+                "kubeflow.org",
+                "v1beta1",
+                "test-ns",
+                "viewers",
+                "viewer-5e1404e679e27b0f0b8ecee8fe515830eaa736c5",
+                V1DeleteOptions {},
+              ]
+            `);
+          done(err);
+        });
+    });
+  });
+});
diff --git a/frontend/src/components/viewers/Tensorboard.test.tsx b/frontend/src/components/viewers/Tensorboard.test.tsx
--- a/frontend/src/components/viewers/Tensorboard.test.tsx
+++ b/frontend/src/components/viewers/Tensorboard.test.tsx
@@ -27,7 +27,14 @@ const DEFAULT_CONFIG: TensorboardViewerConfig = {
   namespace: 'test-ns',
 };
 
-describe('Tensorboard', () => {
+const GET_APP_NOT_FOUND = { podAddress: '', tfVersion: '', image: '' };
+const GET_APP_FOUND = {
+  podAddress: 'podaddress',
+  tfVersion: '1.14.0',
+  image: 'tensorflow/tensorflow:1.14.0',
+};
+
+describe.only('Tensorboard', () => {
   let tree: ReactWrapper | ShallowWrapper;
   const flushPromisesAndTimers = async () => {
     jest.runOnlyPendingTimers();
@@ -50,15 +57,119 @@ describe('Tensorboard', () => {
   });
 
   it('base component snapshot', async () => {
-    const getAppMock = () => Promise.resolve({ podAddress: '', tfVersion: '' });
+    const getAppMock = () => Promise.resolve(GET_APP_NOT_FOUND);
     jest.spyOn(Apis, 'getTensorboardApp').mockImplementation(getAppMock);
     tree = shallow(<TensorboardViewer configs={[]} />);
     await TestUtils.flushPromises();
-    expect(tree).toMatchSnapshot();
+    expect(tree).toMatchInlineSnapshot(`
+      <div>
+        <div>
+          <div
+            className=""
+          >
+            <WithStyles(FormControl)
+              className="formControl"
+            >
+              <WithStyles(WithFormControlContext(InputLabel))
+                htmlFor="viewer-tb-image-select"
+              >
+                TF Image
+              </WithStyles(WithFormControlContext(InputLabel))>
+              <WithStyles(WithFormControlContext(Select))
+                className="select"
+                input={
+                  <WithStyles(Input)
+                    id="viewer-tb-image-select"
+                  />
+                }
+                onChange={[Function]}
+                value="tensorflow/tensorflow:2.2.2"
+              >
+                <WithStyles(ListSubheader)>
+                  Tensoflow 1.x
+                </WithStyles(ListSubheader)>
+                <WithStyles(MenuItem)
+                  value="tensorflow/tensorflow:1.7.1"
+                >
+                  TensorFlow 1.7.1
+                </WithStyles(MenuItem)>
+                <WithStyles(MenuItem)
+                  value="tensorflow/tensorflow:1.8.0"
+                >
+                  TensorFlow 1.8.0
+                </WithStyles(MenuItem)>
+                <WithStyles(MenuItem)
+                  value="tensorflow/tensorflow:1.9.0"
+                >
+                  TensorFlow 1.9.0
+                </WithStyles(MenuItem)>
+                <WithStyles(MenuItem)
+                  value="tensorflow/tensorflow:1.10.1"
+                >
+                  TensorFlow 1.10.1
+                </WithStyles(MenuItem)>
+                <WithStyles(MenuItem)
+                  value="tensorflow/tensorflow:1.11.0"
+                >
+                  TensorFlow 1.11.0
+                </WithStyles(MenuItem)>
+                <WithStyles(MenuItem)
+                  value="tensorflow/tensorflow:1.12.3"
+                >
+                  TensorFlow 1.12.3
+                </WithStyles(MenuItem)>
+                <WithStyles(MenuItem)
+                  value="tensorflow/tensorflow:1.13.2"
+                >
+                  TensorFlow 1.13.2
+                </WithStyles(MenuItem)>
+                <WithStyles(MenuItem)
+                  value="tensorflow/tensorflow:1.14.0"
+                >
+                  TensorFlow 1.14.0
+                </WithStyles(MenuItem)>
+                <WithStyles(MenuItem)
+                  value="tensorflow/tensorflow:1.15.5"
+                >
+                  TensorFlow 1.15.5
+                </WithStyles(MenuItem)>
+                <WithStyles(ListSubheader)>
+                  TensorFlow 2.x
+                </WithStyles(ListSubheader)>
+                <WithStyles(MenuItem)
+                  value="tensorflow/tensorflow:2.0.4"
+                >
+                  TensorFlow 2.0.4
+                </WithStyles(MenuItem)>
+                <WithStyles(MenuItem)
+                  value="tensorflow/tensorflow:2.1.2"
+                >
+                  TensorFlow 2.1.2
+                </WithStyles(MenuItem)>
+                <WithStyles(MenuItem)
+                  value="tensorflow/tensorflow:2.2.2"
+                >
+                  TensorFlow 2.2.2
+                </WithStyles(MenuItem)>
+              </WithStyles(WithFormControlContext(Select))>
+            </WithStyles(FormControl)>
+          </div>
+          <div>
+            <BusyButton
+              busy={false}
+              className="buttonAction"
+              disabled={false}
+              onClick={[Function]}
+              title="Start Tensorboard"
+            />
+          </div>
+        </div>
+      </div>
+    `);
   });
 
   it('does not break on no config', async () => {
-    const getAppMock = () => Promise.resolve({ podAddress: '', tfVersion: '' });
+    const getAppMock = () => Promise.resolve(GET_APP_NOT_FOUND);
     jest.spyOn(Apis, 'getTensorboardApp').mockImplementation(getAppMock);
     tree = shallow(<TensorboardViewer configs={[]} />);
     const base = tree.debug();
@@ -84,7 +195,7 @@ describe('Tensorboard', () => {
   });
 
   it('does not break on empty data', async () => {
-    const getAppMock = () => Promise.resolve({ podAddress: '', tfVersion: '' });
+    const getAppMock = () => Promise.resolve(GET_APP_NOT_FOUND);
     jest.spyOn(Apis, 'getTensorboardApp').mockImplementation(getAppMock);
     const config = { ...DEFAULT_CONFIG, url: '' };
     tree = shallow(<TensorboardViewer configs={[config]} />);
@@ -112,7 +223,11 @@ describe('Tensorboard', () => {
 
   it('shows a link to the tensorboard instance if exists', async () => {
     const config = { ...DEFAULT_CONFIG, url: 'http://test/url' };
-    const getAppMock = () => Promise.resolve({ podAddress: 'test/address', tfVersion: '1.14.0' });
+    const getAppMock = () =>
+      Promise.resolve({
+        ...GET_APP_FOUND,
+        podAddress: 'test/address',
+      });
     jest.spyOn(Apis, 'getTensorboardApp').mockImplementation(getAppMock);
     jest.spyOn(Apis, 'isTensorboardPodReady').mockImplementation(() => Promise.resolve(true));
     tree = shallow(<TensorboardViewer configs={[config]} />);
@@ -121,12 +236,46 @@ describe('Tensorboard', () => {
     await flushPromisesAndTimers();
     expect(Apis.isTensorboardPodReady).toHaveBeenCalledTimes(1);
     expect(Apis.isTensorboardPodReady).toHaveBeenCalledWith('apis/v1beta1/_proxy/test/address');
-    expect(tree).toMatchSnapshot();
+    expect(tree.debug()).toMatchInlineSnapshot(`
+      "<div>
+        <div>
+          <div className=\\"\\">
+            Tensorboard tensorflow/tensorflow:1.14.0 is running for this output.
+          </div>
+          <a href=\\"apis/v1beta1/_proxy/test/address\\" target=\\"_blank\\" rel=\\"noopener noreferrer\\" className=\\"unstyled\\">
+            <WithStyles(Button) className=\\"buttonAction button\\" disabled={false} color=\\"primary\\">
+              Open Tensorboard
+            </WithStyles(Button)>
+          </a>
+          <div>
+            <WithStyles(Button) className=\\"button\\" disabled={false} id=\\"delete\\" title=\\"stop tensorboard and delete its instance\\" onClick={[Function]} color=\\"default\\">
+              Stop Tensorboard
+            </WithStyles(Button)>
+            <WithStyles(Dialog) open={false} onClose={[Function]} aria-labelledby=\\"dialog-title\\">
+              <WithStyles(DialogTitle) id=\\"dialog-title\\">
+                Stop Tensorboard?
+              </WithStyles(DialogTitle)>
+              <WithStyles(DialogContent)>
+                <WithStyles(DialogContentText)>
+                  You can stop the current running tensorboard. The tensorboard viewer will also be deleted from your workloads.
+                </WithStyles(DialogContentText)>
+              </WithStyles(DialogContent)>
+              <WithStyles(DialogActions)>
+                <WithStyles(Button) className=\\"shortButton\\" id=\\"cancel\\" autoFocus={true} onClick={[Function]} color=\\"primary\\">
+                  Cancel
+                </WithStyles(Button)>
+                <BusyButton className=\\"buttonAction shortButton\\" onClick={[Function]} busy={false} color=\\"primary\\" title=\\"Stop\\" />
+              </WithStyles(DialogActions)>
+            </WithStyles(Dialog)>
+          </div>
+        </div>
+      </div>"
+    `);
   });
 
   it('shows start button if no instance exists', async () => {
     const config = DEFAULT_CONFIG;
-    const getAppMock = () => Promise.resolve({ podAddress: '', tfVersion: '' });
+    const getAppMock = () => Promise.resolve(GET_APP_NOT_FOUND);
     const getTensorboardSpy = jest.spyOn(Apis, 'getTensorboardApp').mockImplementation(getAppMock);
     tree = shallow(<TensorboardViewer configs={[DEFAULT_CONFIG]} />);
     const base = tree.debug();
@@ -161,20 +310,24 @@ describe('Tensorboard', () => {
 
   it('starts tensorboard instance when button is clicked', async () => {
     const config = { ...DEFAULT_CONFIG };
-    const getAppMock = () => Promise.resolve({ podAddress: '', tfVersion: '' });
+    const getAppMock = () => Promise.resolve(GET_APP_NOT_FOUND);
     const startAppMock = jest.fn(() => Promise.resolve(''));
     jest.spyOn(Apis, 'getTensorboardApp').mockImplementation(getAppMock);
     jest.spyOn(Apis, 'startTensorboardApp').mockImplementationOnce(startAppMock);
     tree = shallow(<TensorboardViewer configs={[config]} />);
     await TestUtils.flushPromises();
     tree.find('BusyButton').simulate('click');
-    expect(startAppMock).toHaveBeenCalledWith(config.url, '2.0.0', config.namespace);
+    expect(startAppMock).toHaveBeenCalledWith({
+      logdir: config.url,
+      namespace: config.namespace,
+      image: expect.stringContaining('tensorflow/tensorflow:'), // default image
+    });
   });
 
   it('starts tensorboard instance for two configs', async () => {
     const config = { ...DEFAULT_CONFIG, url: 'http://test/url' };
     const config2 = { ...DEFAULT_CONFIG, url: 'http://test/url2' };
-    const getAppMock = jest.fn(() => Promise.resolve({ podAddress: '', tfVersion: '' }));
+    const getAppMock = jest.fn(() => Promise.resolve(GET_APP_NOT_FOUND));
     const startAppMock = jest.fn(() => Promise.resolve(''));
     jest.spyOn(Apis, 'getTensorboardApp').mockImplementation(getAppMock);
     jest.spyOn(Apis, 'startTensorboardApp').mockImplementationOnce(startAppMock);
@@ -186,7 +339,11 @@ describe('Tensorboard', () => {
     );
     tree.find('BusyButton').simulate('click');
     const expectedUrl = `Series1:${config.url},Series2:${config2.url}`;
-    expect(startAppMock).toHaveBeenCalledWith(expectedUrl, '2.0.0', config.namespace);
+    expect(startAppMock).toHaveBeenCalledWith({
+      logdir: expectedUrl,
+      image: expect.stringContaining('tensorflow/tensorflow:'), // default image
+      namespace: config.namespace,
+    });
   });
 
   it('returns friendly display name', () => {
@@ -200,7 +357,7 @@ describe('Tensorboard', () => {
   it('select a version, then start a tensorboard of the corresponding version', async () => {
     const config = { ...DEFAULT_CONFIG };
 
-    const getAppMock = jest.fn(() => Promise.resolve({ podAddress: '', tfVersion: '' }));
+    const getAppMock = jest.fn(() => Promise.resolve(GET_APP_NOT_FOUND));
     const startAppMock = jest.fn(() => Promise.resolve(''));
     jest.spyOn(Apis, 'getTensorboardApp').mockImplementation(getAppMock);
     const startAppSpy = jest
@@ -215,18 +372,20 @@ describe('Tensorboard', () => {
       .find('[role="button"]')
       .simulate('click');
     tree
-      .findWhere(el => el.text() === 'TensorFlow 1.15.0')
+      .findWhere(el => el.text().startsWith('TensorFlow 1.15'))
       .hostNodes()
       .simulate('click');
     tree.find('BusyButton').simulate('click');
-    expect(startAppSpy).toHaveBeenCalledWith(config.url, '1.15.0', config.namespace);
+    expect(startAppSpy).toHaveBeenCalledWith({
+      logdir: config.url,
+      image: 'tensorflow/tensorflow:1.15.5',
+      namespace: config.namespace,
+    });
   });
 
   it('delete the tensorboard instance, confirm in the dialog,\
     then return back to previous page', async () => {
-    const getAppMock = jest.fn(() =>
-      Promise.resolve({ podAddress: 'podaddress', tfVersion: '1.14.0' }),
-    );
+    const getAppMock = jest.fn(() => Promise.resolve(GET_APP_FOUND));
     jest.spyOn(Apis, 'getTensorboardApp').mockImplementation(getAppMock);
     const deleteAppMock = jest.fn(() => Promise.resolve(''));
     const deleteAppSpy = jest.spyOn(Apis, 'deleteTensorboardApp').mockImplementation(deleteAppMock);
@@ -252,9 +411,7 @@ describe('Tensorboard', () => {
 
   it('show version info in delete confirming dialog, \
     if a tensorboard instance already exists', async () => {
-    const getAppMock = jest.fn(() =>
-      Promise.resolve({ podAddress: 'podaddress', tfVersion: '1.14.0' }),
-    );
+    const getAppMock = jest.fn(() => Promise.resolve(GET_APP_FOUND));
     jest.spyOn(Apis, 'getTensorboardApp').mockImplementation(getAppMock);
     const config = DEFAULT_CONFIG;
     tree = mount(<TensorboardViewer configs={[config]} />);
@@ -264,13 +421,11 @@ describe('Tensorboard', () => {
       .find('#delete')
       .find('Button')
       .simulate('click');
-    expect(tree.findWhere(el => el.text() === 'Stop Tensorboard 1.14.0?').exists()).toBeTruthy();
+    expect(tree.findWhere(el => el.text() === 'Stop Tensorboard?').exists()).toBeTruthy();
   });
 
   it('click on cancel on delete tensorboard dialog, then return back to previous page', async () => {
-    const getAppMock = jest.fn(() =>
-      Promise.resolve({ podAddress: 'podaddress', tfVersion: '1.14.0' }),
-    );
+    const getAppMock = jest.fn(() => Promise.resolve(GET_APP_FOUND));
     jest.spyOn(Apis, 'getTensorboardApp').mockImplementation(getAppMock);
     const config = DEFAULT_CONFIG;
     tree = mount(<TensorboardViewer configs={[config]} />);
@@ -287,13 +442,11 @@ describe('Tensorboard', () => {
       .simulate('click');
 
     expect(tree.findWhere(el => el.text() === 'Open Tensorboard').exists()).toBeTruthy();
-    expect(tree.findWhere(el => el.text() === 'Delete Tensorboard').exists()).toBeTruthy();
+    expect(tree.findWhere(el => el.text() === 'Stop Tensorboard').exists()).toBeTruthy();
   });
 
   it('asks user to wait when Tensorboard status is not ready', async () => {
-    const getAppMock = jest.fn(() =>
-      Promise.resolve({ podAddress: 'podaddress', tfVersion: '1.14.0' }),
-    );
+    const getAppMock = jest.fn(() => Promise.resolve(GET_APP_FOUND));
     jest.spyOn(Apis, 'getTensorboardApp').mockImplementation(getAppMock);
     jest.spyOn(Apis, 'isTensorboardPodReady').mockImplementation(() => Promise.resolve(false));
     jest.spyOn(Apis, 'deleteTensorboardApp').mockImplementation(jest.fn(() => Promise.resolve('')));
@@ -314,7 +467,7 @@ describe('Tensorboard', () => {
         )
         .exists(),
     ).toBeTruthy();
-    expect(tree.findWhere(el => el.text() === 'Delete Tensorboard').exists()).toBeTruthy();
+    expect(tree.findWhere(el => el.text() === 'Stop Tensorboard').exists()).toBeTruthy();
 
     // After a while, it is ready and wait message is not shwon any more
     jest.spyOn(Apis, 'isTensorboardPodReady').mockImplementation(() => Promise.resolve(true));
diff --git a/frontend/src/components/viewers/__snapshots__/Tensorboard.test.tsx.snap b/frontend/src/components/viewers/__snapshots__/Tensorboard.test.tsx.snap
deleted file mode 100644
--- a/frontend/src/components/viewers/__snapshots__/Tensorboard.test.tsx.snap
+++ /dev/null
@@ -1,184 +0,0 @@
-// Jest Snapshot v1, https://goo.gl/fbAQLP
-
-exports[`Tensorboard base component snapshot 1`] = `
-<div>
-  <div>
-    <div
-      className=""
-    >
-      <WithStyles(FormControl)
-        className="formControl"
-      >
-        <WithStyles(WithFormControlContext(InputLabel))
-          htmlFor="grouped-select"
-        >
-          TF Version
-        </WithStyles(WithFormControlContext(InputLabel))>
-        <WithStyles(WithFormControlContext(Select))
-          className="select"
-          input={
-            <WithStyles(Input)
-              id="grouped-select"
-            />
-          }
-          onChange={[Function]}
-          value="2.0.0"
-        >
-          <WithStyles(ListSubheader)>
-            Tensoflow 1.x
-          </WithStyles(ListSubheader)>
-          <WithStyles(MenuItem)
-            value="1.4.0"
-          >
-            TensorFlow 1.4.0
-          </WithStyles(MenuItem)>
-          <WithStyles(MenuItem)
-            value="1.5.0"
-          >
-            TensorFlow 1.5.0
-          </WithStyles(MenuItem)>
-          <WithStyles(MenuItem)
-            value="1.6.0"
-          >
-            TensorFlow 1.6.0
-          </WithStyles(MenuItem)>
-          <WithStyles(MenuItem)
-            value="1.7.0"
-          >
-            TensorFlow 1.7.0
-          </WithStyles(MenuItem)>
-          <WithStyles(MenuItem)
-            value="1.8.0"
-          >
-            TensorFlow 1.8.0
-          </WithStyles(MenuItem)>
-          <WithStyles(MenuItem)
-            value="1.9.0"
-          >
-            TensorFlow 1.9.0
-          </WithStyles(MenuItem)>
-          <WithStyles(MenuItem)
-            value="1.10.0"
-          >
-            TensorFlow 1.10.0
-          </WithStyles(MenuItem)>
-          <WithStyles(MenuItem)
-            value="1.11.0"
-          >
-            TensorFlow 1.11.0
-          </WithStyles(MenuItem)>
-          <WithStyles(MenuItem)
-            value="1.12.0"
-          >
-            TensorFlow 1.12.0
-          </WithStyles(MenuItem)>
-          <WithStyles(MenuItem)
-            value="1.13.2"
-          >
-            TensorFlow 1.13.2
-          </WithStyles(MenuItem)>
-          <WithStyles(MenuItem)
-            value="1.14.0"
-          >
-            TensorFlow 1.14.0
-          </WithStyles(MenuItem)>
-          <WithStyles(MenuItem)
-            value="1.15.0"
-          >
-            TensorFlow 1.15.0
-          </WithStyles(MenuItem)>
-          <WithStyles(ListSubheader)>
-            TensorFlow 2.x
-          </WithStyles(ListSubheader)>
-          <WithStyles(MenuItem)
-            value="2.0.0"
-          >
-            TensorFlow 2.0.0
-          </WithStyles(MenuItem)>
-        </WithStyles(WithFormControlContext(Select))>
-      </WithStyles(FormControl)>
-    </div>
-    <div>
-      <BusyButton
-        busy={false}
-        className="buttonAction"
-        disabled={false}
-        onClick={[Function]}
-        title="Start Tensorboard"
-      />
-    </div>
-  </div>
-</div>
-`;
-
-exports[`Tensorboard shows a link to the tensorboard instance if exists 1`] = `
-<div>
-  <div>
-    <div
-      className=""
-    >
-      Tensorboard 1.14.0 is running for this output.
-    </div>
-    <a
-      className="unstyled"
-      href="apis/v1beta1/_proxy/test/address"
-      rel="noopener noreferrer"
-      target="_blank"
-    >
-      <WithStyles(Button)
-        className="buttonAction button"
-        color="primary"
-        disabled={false}
-      >
-        Open Tensorboard
-      </WithStyles(Button)>
-    </a>
-    <div>
-      <WithStyles(Button)
-        className="button"
-        color="default"
-        disabled={false}
-        id="delete"
-        onClick={[Function]}
-        title="stop tensorboard and delete its instance"
-      >
-        Delete Tensorboard
-      </WithStyles(Button)>
-      <WithStyles(Dialog)
-        aria-labelledby="dialog-title"
-        onClose={[Function]}
-        open={false}
-      >
-        <WithStyles(DialogTitle)
-          id="dialog-title"
-        >
-          Stop Tensorboard 1.14.0?
-        </WithStyles(DialogTitle)>
-        <WithStyles(DialogContent)>
-          <WithStyles(DialogContentText)>
-            You can stop the current running tensorboard. The tensorboard viewer will also be deleted from your workloads.
-          </WithStyles(DialogContentText)>
-        </WithStyles(DialogContent)>
-        <WithStyles(DialogActions)>
-          <WithStyles(Button)
-            autoFocus={true}
-            className="shortButton"
-            color="primary"
-            id="cancel"
-            onClick={[Function]}
-          >
-            Cancel
-          </WithStyles(Button)>
-          <BusyButton
-            busy={false}
-            className="buttonAction shortButton"
-            color="primary"
-            onClick={[Function]}
-            title="Stop"
-          />
-        </WithStyles(DialogActions)>
-      </WithStyles(Dialog)>
-    </div>
-  </div>
-</div>
-`;
diff --git a/frontend/src/lib/Apis.test.ts b/frontend/src/lib/Apis.test.ts
--- a/frontend/src/lib/Apis.test.ts
+++ b/frontend/src/lib/Apis.test.ts
@@ -166,30 +166,109 @@ describe('Apis', () => {
 
   it('getTensorboardApp', async () => {
     const spy = fetchSpy(
-      JSON.stringify({ podAddress: 'http://some/address', tfVersion: '1.14.0' }),
+      JSON.stringify({
+        podAddress: 'http://some/address',
+        tfVersion: '1.14.0',
+        image: 'tensorflow/tensorflow:1.14.0',
+      }),
     );
     const tensorboardInstance = await Apis.getTensorboardApp('gs://log/dir', 'test-ns');
-    expect(tensorboardInstance).toEqual({ podAddress: 'http://some/address', tfVersion: '1.14.0' });
+    expect(tensorboardInstance).toEqual({
+      podAddress: 'http://some/address',
+      tfVersion: '1.14.0',
+      image: 'tensorflow/tensorflow:1.14.0',
+    });
     expect(spy).toHaveBeenCalledWith(
       `apps/tensorboard?logdir=${encodeURIComponent('gs://log/dir')}&namespace=test-ns`,
       { credentials: 'same-origin' },
     );
   });
 
-  it('startTensorboardApp', async () => {
-    const spy = fetchSpy('http://some/address');
-    await Apis.startTensorboardApp('gs://log/dir', '1.14.0', 'test-ns');
-    expect(spy).toHaveBeenCalledWith(
-      'apps/tensorboard?logdir=' +
-        encodeURIComponent('gs://log/dir') +
-        '&tfversion=1.14.0' +
-        '&namespace=test-ns',
-      {
-        credentials: 'same-origin',
-        headers: { 'content-type': 'application/json' },
-        method: 'POST',
-      },
-    );
+  describe('startTensorboardApp', () => {
+    const defaultArgs = {
+      logdir: 'gs://log/dir',
+      image: 'tensorflow/tensorflow:1.14.0',
+      namespace: 'test-ns',
+    };
+    it('starts tensorboard app', async () => {
+      const spy = fetchSpy('http://some/address');
+      await Apis.startTensorboardApp(defaultArgs);
+      expect(spy).toHaveBeenCalledWith(
+        'apps/tensorboard?logdir=' +
+          encodeURIComponent(defaultArgs.logdir) +
+          '&namespace=' +
+          defaultArgs.namespace +
+          '&image=' +
+          encodeURIComponent(defaultArgs.image),
+        {
+          credentials: 'same-origin',
+          headers: { 'content-type': 'application/json' },
+          method: 'POST',
+        },
+      );
+    });
+    it('encodes podTemplateSpec as JSON in arg', async () => {
+      const spy = fetchSpy('http://some/address');
+      const args = {
+        ...defaultArgs,
+        podTemplateSpec: {
+          spec: {
+            containers: [
+              {
+                env: [
+                  {
+                    name: 'AWS_ACCESS_KEY_ID',
+                    valueFrom: {
+                      secretKeyRef: {
+                        name: 'mlpipeline-minio-artifact',
+                        key: 'accesskey',
+                      },
+                    },
+                  },
+                  {
+                    name: 'AWS_SECRET_ACCESS_KEY',
+                    valueFrom: {
+                      secretKeyRef: {
+                        name: 'mlpipeline-minio-artifact',
+                        key: 'secretkey',
+                      },
+                    },
+                  },
+                  {
+                    name: 'AWS_REGION',
+                    value: 'minio',
+                  },
+                  {
+                    name: 'S3_ENDPOINT',
+                    value: 'http://minio-service:9000',
+                  },
+                  {
+                    name: 'S3_USE_HTTPS',
+                    value: '0',
+                  },
+                  {
+                    name: 'S3_VERIFY_SSL',
+                    value: '0',
+                  },
+                ],
+              },
+            ],
+          },
+        },
+      };
+      await Apis.startTensorboardApp(args);
+      expect(spy).toHaveBeenCalledWith(
+        'apps/tensorboard?logdir=' +
+          encodeURIComponent(args.logdir) +
+          '&namespace=' +
+          args.namespace +
+          '&image=' +
+          encodeURIComponent(args.image) +
+          '&podtemplatespec=' +
+          encodeURIComponent(JSON.stringify(args.podTemplateSpec)),
+        expect.anything(),
+      );
+    });
   });
 
   it('deleteTensorboardApp', async () => {
diff --git a/samples/core/XGBoost/xgboost_sample_test.py b/samples/core/XGBoost/xgboost_sample_test.py
--- a/samples/core/XGBoost/xgboost_sample_test.py
+++ b/samples/core/XGBoost/xgboost_sample_test.py
@@ -17,22 +17,6 @@
 from ...test.util import run_pipeline_func, TestCase, NEEDS_A_FIX
 
 run_pipeline_func([
-    # TODO(v2-compatible): fix this sample for v2 mode.
-    # V2_COMPATIBLE mode fails with:
-    # File "/Users/gongyuan/kfp/pipelines/sdk/python/kfp/compiler/v2_compat.py", line 108, in update_op
-    # artifact_info = {"fileInputPath": op.input_artifact_paths[artifact_name]}
-    # KeyError: 'text'
-    #
-    # And another error:
-    # This step is in Error state with this message: withParam value could not
-    # be parsed as a JSON list:
-    # {
-    #   "id":"733",
-    #   "typeId":"114",
-    #   "uri":"gs://gongyuan-test/kfp/output/pipeline-with-loop-parameter/pipeline-with-loop-parameter-pn8nj/pipeline-with-loop-parameter-pn8nj-1197421549/data",
-    #   "createTimeSinceEpoch":"1617689193656",
-    #   "lastUpdateTimeSinceEpoch":"1617689193656"
-    # }
     TestCase(
         pipeline_func=xgboost_pipeline,
         mode=kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE,
diff --git a/samples/core/visualization/tensorboard_minio_test.py b/samples/core/visualization/tensorboard_minio_test.py
new file mode 100644
--- /dev/null
+++ b/samples/core/visualization/tensorboard_minio_test.py
@@ -0,0 +1,24 @@
+# Copyright 2021 The Kubeflow Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import kfp
+from .tensorboard_minio import my_pipeline
+from ...test.util import run_pipeline_func, TestCase
+
+run_pipeline_func([
+    TestCase(
+        pipeline_func=my_pipeline,
+        mode=kfp.dsl.PipelineExecutionMode.V1_LEGACY,
+    ),
+])
diff --git a/samples/test/config.yaml b/samples/test/config.yaml
--- a/samples/test/config.yaml
+++ b/samples/test/config.yaml
@@ -59,6 +59,8 @@
   path: samples.test.various_io_types
 - name: simple_pipeline_without_importer
   path: samples.test.simple_pipeline_without_importer_test
+- name: tensorboard_minio
+  path: samples.core.visualization.tensorboard_minio_test
 - name: lightweight_python_functions_v2_with_outputs
   path: samples.test.lightweight_python_functions_v2_with_outputs_test
 - name: lightweight_python_functions_v2_pipeline
diff --git a/samples/test/util.py b/samples/test/util.py
--- a/samples/test/util.py
+++ b/samples/test/util.py
@@ -15,6 +15,7 @@
 import json
 import logging
 import os
+import time
 import random
 from dataclasses import dataclass, asdict
 from pprint import pprint
@@ -31,9 +32,9 @@
 # Add **kwargs, so that when new arguments are added, this doesn't fail for
 # unknown arguments.
 def _default_verify_func(
-        run_id: int, run: kfp_server_api.ApiRun,
-        mlmd_connection_config: metadata_store_pb2.MetadataStoreClientConfig,
-        **kwargs
+    run_id: int, run: kfp_server_api.ApiRun,
+    mlmd_connection_config: metadata_store_pb2.MetadataStoreClientConfig,
+    **kwargs
 ):
     assert run.status == 'Succeeded'
 
@@ -50,9 +51,9 @@ class TestCase:
     mode: kfp.dsl.PipelineExecutionMode = kfp.dsl.PipelineExecutionMode.V2_COMPATIBLE
     arguments: Optional[Dict[str, str]] = None
     verify_func: Callable[[
-                              int, kfp_server_api.ApiRun, kfp_server_api.
-            ApiRunDetail, metadata_store_pb2.MetadataStoreClientConfig
-                          ], None] = _default_verify_func
+        int, kfp_server_api.ApiRun, kfp_server_api.
+        ApiRunDetail, metadata_store_pb2.MetadataStoreClientConfig
+    ], None] = _default_verify_func
 
 
 def run_pipeline_func(test_cases: List[TestCase]):
@@ -63,9 +64,9 @@ def run_pipeline_func(test_cases: List[TestCase]):
     """
 
     def test_wrapper(
-            run_pipeline: Callable[[Callable, kfp.dsl.PipelineExecutionMode, dict],
-                                   kfp_server_api.ApiRunDetail],
-            mlmd_connection_config: metadata_store_pb2.MetadataStoreClientConfig,
+        run_pipeline: Callable[[Callable, kfp.dsl.PipelineExecutionMode, dict],
+                               kfp_server_api.ApiRunDetail],
+        mlmd_connection_config: metadata_store_pb2.MetadataStoreClientConfig,
     ):
         for case in test_cases:
             run_detail = run_pipeline(
@@ -94,26 +95,28 @@ def _retry_with_backoff(fn: Callable, retries=5, backoff_in_seconds=1):
     while True:
         try:
             return fn()
-        except:
+        except Exception as e:
             if i >= retries:
-                print(f"Failed after {retires} retries:")
+                print(f"Failed after {retries} retries:")
                 raise
             else:
-                sleep = (backoff_in_seconds * 2 ** i + random.uniform(0, 1))
-                print("  Sleep :", str(sleep) + "s")
+                print(e)
+                sleep = (backoff_in_seconds * 2**i + random.uniform(0, 1))
+                print("  Retry after ", str(sleep) + "s")
                 time.sleep(sleep)
                 i += 1
 
 
 def _run_test(callback):
+
     def main(
-            output_directory: Optional[str] = None,  # example
-            host: Optional[str] = None,
-            external_host: Optional[str] = None,
-            launcher_image: Optional['URI'] = None,
-            experiment: str = 'v2_sample_test_samples',
-            metadata_service_host: Optional[str] = None,
-            metadata_service_port: int = 8080,
+        output_directory: Optional[str] = None,  # example
+        host: Optional[str] = None,
+        external_host: Optional[str] = None,
+        launcher_image: Optional['URI'] = None,
+        experiment: str = 'v2_sample_test_samples',
+        metadata_service_host: Optional[str] = None,
+        metadata_service_port: int = 8080,
     ):
         """Test file CLI entrypoint used by Fire.
 
@@ -152,10 +155,10 @@ def main(
         client = kfp.Client(host=host)
 
         def run_pipeline(
-                pipeline_func: Callable,
-                mode: kfp.dsl.PipelineExecutionMode = kfp.dsl.PipelineExecutionMode.
-                    V2_COMPATIBLE,
-                arguments: dict = {},
+            pipeline_func: Callable,
+            mode: kfp.dsl.PipelineExecutionMode = kfp.dsl.PipelineExecutionMode.
+            V2_COMPATIBLE,
+            arguments: dict = {},
         ) -> kfp_server_api.ApiRunDetail:
             extra_arguments = {}
             if mode != kfp.dsl.PipelineExecutionMode.V1_LEGACY:
@@ -223,8 +226,8 @@ class KfpArtifact:
 
     @classmethod
     def new(
-            cls, mlmd_artifact: metadata_store_pb2.Artifact,
-            mlmd_artifact_type: metadata_store_pb2.ArtifactType
+        cls, mlmd_artifact: metadata_store_pb2.Artifact,
+        mlmd_artifact_type: metadata_store_pb2.ArtifactType
     ):
         custom_properties = {}
         for k, v in mlmd_artifact.custom_properties.items():
@@ -281,13 +284,13 @@ def get_dict(self):
 
     @classmethod
     def new(
-            cls,
-            context: metadata_store_pb2.Context,
-            execution: metadata_store_pb2.Execution,
-            execution_types_by_id,  # dict[int, metadata_store_pb2.ExecutionType]
-            events_by_execution_id,  # dict[int, List[metadata_store_pb2.Event]]
-            artifacts_by_id,  # dict[int, metadata_store_pb2.Artifact]
-            artifact_types_by_id,  # dict[int, metadata_store_pb2.ArtifactType]
+        cls,
+        context: metadata_store_pb2.Context,
+        execution: metadata_store_pb2.Execution,
+        execution_types_by_id,  # dict[int, metadata_store_pb2.ExecutionType]
+        events_by_execution_id,  # dict[int, List[metadata_store_pb2.Event]]
+        artifacts_by_id,  # dict[int, metadata_store_pb2.Artifact]
+        artifact_types_by_id,  # dict[int, metadata_store_pb2.ArtifactType]
     ):
         execution_type = execution_types_by_id[execution.type_id]
         params = _parse_parameters(execution)
@@ -336,7 +339,8 @@ class KfpMlmdClient:
 
     def __init__(
         self,
-        mlmd_connection_config: Optional[metadata_store_pb2.MetadataStoreClientConfig] = None,
+        mlmd_connection_config: Optional[
+            metadata_store_pb2.MetadataStoreClientConfig] = None,
     ):
         if mlmd_connection_config is None:
             # default to value suitable for local testing