Repository: kubeflow/pipelines

Issue:
pipline api-server did't print  error log
### What steps did you take:
[A clear and concise description of what the bug is.]

I use kustomize  deploy kubeflow 
### What happened:
api-server  never output error log ,but pod crashbackoof 
### What did you expect to happen:
api-server output log tell me what happend
### Environment:

<!-- Please fill in those that seem relevant. -->
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.4", GitCommit:"67d2fcf276fcd9cf743ad4be9a9ef5828adc082f", GitTreeState:"clean", BuildDate:"2019-09-18T14:41:55Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

How did you deploy Kubeflow Pipelines (KFP)?
<!-- If you are not sure, here's [an introduction of all options](https://www.kubeflow.org/docs/pipelines/installation/overview/). -->

KFP version: <!-- If you are not sure, build commit shows on bottom of KFP UI left sidenav. -->

KFP SDK version: <!-- Please attach the output of this shell command: $pip list | grep kfp -->


### Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

/kind bug
<!-- Please include labels by uncommenting them to help us better triage issues, choose from the following -->
<!--

/area backend

// /area engprod
-->

flexible pipeline service (host) path in client SDK 
when creating an SDK `Client()` the path to `ml-pipeline` API service is loaded from a hard coded value (`ml-pipeline.kubeflow.svc.cluster.local:8888`) which indicate a specific k8s namespace. it can be valuable to load that default value from an env variable, i.e. changing the line in `_client.py` from:

`config.host = host if host else Client.IN_CLUSTER_DNS_NAME`

to:

`config.host = host or os.environ.get('ML_PIPELINE_DNS_NAME',Client.IN_CLUSTER_DNS_NAME)`

also note that when a user provide the `host` parameter, the ipython output points to the API server and not to the UI service (see the logic in `_get_url_prefix()`), it seems like a potential bug

if its acceptable i can submit a PR for the line change above
 


Discussion:

>it can be valuable to load that default value from an env variable

I'm not sure about the user scenarios, but I guess it might be useful for some cases.

>ML_PIPELINE_DNS_NAME

Maybe, `KF_PIPELINES_ENDPOINT`?

>it seems like a potential bug

It probably is. The Client class can definitely use some improvements, especially for out-of-cluster usage.

> if its acceptable i can submit a PR

We always appreciate PRs.
Fixed in https://github.com/kubeflow/pipelines/pull/1293

Resolving patch:
diff --git a/sdk/python/kfp/_client.py b/sdk/python/kfp/_client.py
--- a/sdk/python/kfp/_client.py
+++ b/sdk/python/kfp/_client.py
@@ -90,7 +90,7 @@ class Client(object):
   LOCAL_KFP_CONTEXT = os.path.expanduser('~/.config/kfp/context.json')
 
   # TODO: Wrap the configurations for different authentication methods.
-  def __init__(self, host=None, client_id=None, namespace='kubeflow', other_client_id=None, other_client_secret=None, existing_token=None, cookies=None):
+  def __init__(self, host=None, client_id=None, namespace='kubeflow', other_client_id=None, other_client_secret=None, existing_token=None, cookies=None, proxy=None, ssl_ca_cert=None):
     """Create a new instance of kfp client.
 
     Args:
@@ -109,10 +109,12 @@ def __init__(self, host=None, client_id=None, namespace='kubeflow', other_client
       existing_token: pass in token directly, it's used for cases better get token outside of SDK, e.x. GCP Cloud Functions
           or caller already has a token
       cookies: CookieJar object containing cookies that will be passed to the pipelines API.
+      proxy: HTTP or HTTPS proxy server
+      ssl_ca_cert: cert for proxy
     """
     host = host or os.environ.get(KF_PIPELINES_ENDPOINT_ENV)
     self._uihost = os.environ.get(KF_PIPELINES_UI_ENDPOINT_ENV, host)
-    config = self._load_config(host, client_id, namespace, other_client_id, other_client_secret, existing_token)
+    config = self._load_config(host, client_id, namespace, other_client_id, other_client_secret, existing_token, proxy, ssl_ca_cert)
     # Save the loaded API client configuration, as a reference if update is
     # needed.
     self._existing_config = config
@@ -125,14 +127,21 @@ def __init__(self, host=None, client_id=None, namespace='kubeflow', other_client
     self._upload_api = kfp_server_api.api.PipelineUploadServiceApi(api_client)
     self._load_context_setting_or_default()
 
-  def _load_config(self, host, client_id, namespace, other_client_id, other_client_secret, existing_token):
+  def _load_config(self, host, client_id, namespace, other_client_id, other_client_secret, existing_token, proxy, ssl_ca_cert):
     config = kfp_server_api.configuration.Configuration()
 
+    if proxy:
+      # https://github.com/kubeflow/pipelines/blob/c6ac5e0b1fd991e19e96419f0f508ec0a4217c29/backend/api/python_http_client/kfp_server_api/rest.py#L100
+      config.proxy = proxy
+
+    if ssl_ca_cert:
+      config.ssl_ca_cert = ssl_ca_cert
+
     host = host or ''
     # Preprocess the host endpoint to prevent some common user mistakes.
-    # This should only be done for non-IAP cases (when client_id is None). IAP requires preserving the protocol.
     if not client_id:
-      host = re.sub(r'^(http|https)://', '', host).rstrip('/')
+      # always preserving the protocol (http://localhost requires it)
+      host = host.rstrip('/')
 
     if host:
       config.host = host


Test patch:
diff --git a/backend/src/agent/persistence/worker/metrics_reporter_test.go b/backend/src/agent/persistence/worker/metrics_reporter_test.go
--- a/backend/src/agent/persistence/worker/metrics_reporter_test.go
+++ b/backend/src/agent/persistence/worker/metrics_reporter_test.go
@@ -51,6 +51,32 @@ func TestReportMetrics_NoCompletedNode_NoOP(t *testing.T) {
 	assert.Nil(t, pipelineFake.GetReportedMetricsRequest())
 }
 
+func TestReportMetrics_NoRunID_NoOP(t *testing.T) {
+	pipelineFake := client.NewPipelineClientFake()
+
+	reporter := NewMetricsReporter(pipelineFake)
+
+	workflow := util.NewWorkflow(&workflowapi.Workflow{
+		ObjectMeta: metav1.ObjectMeta{
+			Namespace: "MY_NAMESPACE",
+			Name:      "MY_NAME",
+			UID:       types.UID("run-1"),
+		},
+		Status: workflowapi.WorkflowStatus{
+			Nodes: map[string]workflowapi.NodeStatus{
+				"node-1": workflowapi.NodeStatus{
+					ID:    "node-1",
+					Phase: workflowapi.NodeSucceeded,
+				},
+			},
+		},
+	})
+	err := reporter.ReportMetrics(workflow)
+	assert.Nil(t, err)
+	assert.Nil(t, pipelineFake.GetReadArtifactRequest())
+	assert.Nil(t, pipelineFake.GetReportedMetricsRequest())
+}
+
 func TestReportMetrics_NoArtifact_NoOP(t *testing.T) {
 	pipelineFake := client.NewPipelineClientFake()
 
@@ -61,18 +87,50 @@ func TestReportMetrics_NoArtifact_NoOP(t *testing.T) {
 			Namespace: "MY_NAMESPACE",
 			Name:      "MY_NAME",
 			UID:       types.UID("run-1"),
+			Labels:    map[string]string{util.LabelKeyWorkflowRunId: "run-1"},
+		},
+		Status: workflowapi.WorkflowStatus{
+			Nodes: map[string]workflowapi.NodeStatus{
+				"node-1": workflowapi.NodeStatus{
+					ID:    "node-1",
+					Phase: workflowapi.NodeSucceeded,
+				},
+			},
+		},
+	})
+	err := reporter.ReportMetrics(workflow)
+	assert.Nil(t, err)
+	assert.Nil(t, pipelineFake.GetReadArtifactRequest())
+	assert.Nil(t, pipelineFake.GetReportedMetricsRequest())
+}
+
+func TestReportMetrics_NoMetricsArtifact_NoOP(t *testing.T) {
+	pipelineFake := client.NewPipelineClientFake()
+
+	reporter := NewMetricsReporter(pipelineFake)
+
+	workflow := util.NewWorkflow(&workflowapi.Workflow{
+		ObjectMeta: metav1.ObjectMeta{
+			Namespace: "MY_NAMESPACE",
+			Name:      "MY_NAME",
+			UID:       types.UID("run-1"),
+			Labels:    map[string]string{util.LabelKeyWorkflowRunId: "run-1"},
 		},
 		Status: workflowapi.WorkflowStatus{
 			Nodes: map[string]workflowapi.NodeStatus{
 				"node-1": workflowapi.NodeStatus{
 					ID:    "node-1",
 					Phase: workflowapi.NodeSucceeded,
+					Outputs: &workflowapi.Outputs{
+						Artifacts: []workflowapi.Artifact{{Name: "mlpipeline-ui-metadata"}},
+					},
 				},
 			},
 		},
 	})
 	err := reporter.ReportMetrics(workflow)
 	assert.Nil(t, err)
+	assert.Nil(t, pipelineFake.GetReadArtifactRequest())
 	assert.Nil(t, pipelineFake.GetReportedMetricsRequest())
 }
 
@@ -91,6 +149,9 @@ func TestReportMetrics_Succeed(t *testing.T) {
 				"node-1": workflowapi.NodeStatus{
 					ID:    "node-1",
 					Phase: workflowapi.NodeSucceeded,
+					Outputs: &workflowapi.Outputs{
+						Artifacts: []workflowapi.Artifact{{Name: "mlpipeline-metrics"}},
+					},
 				},
 			},
 		},
@@ -146,6 +207,9 @@ func TestReportMetrics_EmptyArchive_Fail(t *testing.T) {
 				"node-1": workflowapi.NodeStatus{
 					ID:    "node-1",
 					Phase: workflowapi.NodeSucceeded,
+					Outputs: &workflowapi.Outputs{
+						Artifacts: []workflowapi.Artifact{{Name: "mlpipeline-metrics"}},
+					},
 				},
 			},
 		},
@@ -184,6 +248,9 @@ func TestReportMetrics_MultipleFilesInArchive_Fail(t *testing.T) {
 				"node-1": workflowapi.NodeStatus{
 					ID:    "node-1",
 					Phase: workflowapi.NodeSucceeded,
+					Outputs: &workflowapi.Outputs{
+						Artifacts: []workflowapi.Artifact{{Name: "mlpipeline-metrics"}},
+					},
 				},
 			},
 		},
@@ -224,6 +291,9 @@ func TestReportMetrics_InvalidMetricsJSON_Fail(t *testing.T) {
 				"node-1": workflowapi.NodeStatus{
 					ID:    "node-1",
 					Phase: workflowapi.NodeSucceeded,
+					Outputs: &workflowapi.Outputs{
+						Artifacts: []workflowapi.Artifact{{Name: "mlpipeline-metrics"}},
+					},
 				},
 			},
 		},
@@ -263,10 +333,16 @@ func TestReportMetrics_InvalidMetricsJSON_PartialFail(t *testing.T) {
 				"node-1": workflowapi.NodeStatus{
 					ID:    "node-1",
 					Phase: workflowapi.NodeSucceeded,
+					Outputs: &workflowapi.Outputs{
+						Artifacts: []workflowapi.Artifact{{Name: "mlpipeline-metrics"}},
+					},
 				},
 				"node-2": workflowapi.NodeStatus{
 					ID:    "node-2",
 					Phase: workflowapi.NodeSucceeded,
+					Outputs: &workflowapi.Outputs{
+						Artifacts: []workflowapi.Artifact{{Name: "mlpipeline-metrics"}},
+					},
 				},
 			},
 		},
@@ -333,6 +409,9 @@ func TestReportMetrics_CorruptedArchiveFile_Fail(t *testing.T) {
 				"node-1": workflowapi.NodeStatus{
 					ID:    "node-1",
 					Phase: workflowapi.NodeSucceeded,
+					Outputs: &workflowapi.Outputs{
+						Artifacts: []workflowapi.Artifact{{Name: "mlpipeline-metrics"}},
+					},
 				},
 			},
 		},
@@ -370,12 +449,15 @@ func TestReportMetrics_MultiplMetricErrors_TransientErrowWin(t *testing.T) {
 				"node-1": workflowapi.NodeStatus{
 					ID:    "node-1",
 					Phase: workflowapi.NodeSucceeded,
+					Outputs: &workflowapi.Outputs{
+						Artifacts: []workflowapi.Artifact{{Name: "mlpipeline-metrics"}},
+					},
 				},
 			},
 		},
 	})
 	metricsJSON :=
-			`{"metrics": [{"name": "accuracy", "numberValue": 0.77}, {"name": "log loss", "numberValue": 1.2}, {"name": "accuracy", "numberValue": 1.2}]}`
+		`{"metrics": [{"name": "accuracy", "numberValue": 0.77}, {"name": "log loss", "numberValue": 1.2}, {"name": "accuracy", "numberValue": 1.2}]}`
 	artifactData, _ := util.ArchiveTgz(map[string]string{"file": metricsJSON})
 	pipelineFake.StubArtifact(
 		&api.ReadArtifactRequest{
diff --git a/test/deploy-cluster.sh b/test/deploy-cluster.sh
--- a/test/deploy-cluster.sh
+++ b/test/deploy-cluster.sh
@@ -72,7 +72,7 @@ else
     --enable-autoscaling --max-nodes=8 --min-nodes=2"
   KUBERNETES_VERSION_ARG="--cluster-version=1.14"
   if [ "$ENABLE_WORKLOAD_IDENTITY" = true ]; then
-    WI_ARG="--identity-namespace=$PROJECT.svc.id.goog"
+    WI_ARG="--workload-pool=$PROJECT.svc.id.goog"
     SCOPE_ARG=
   else
     WI_ARG=
diff --git a/test/presubmit-backend-visualization.sh b/test/presubmit-backend-visualization.sh
new file mode 100755
--- /dev/null
+++ b/test/presubmit-backend-visualization.sh
@@ -0,0 +1,23 @@
+#!/bin/bash -ex
+# Copyright 2020 Kubeflow Pipelines contributors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+source_root=$(pwd)
+
+# Visualization test
+cd "$source_root/backend/src/apiserver/visualization"
+python3 -m pip install --upgrade pip
+python3 -m pip install -r requirements-test.txt
+python3 test_exporter.py
+python3 test_server.py
diff --git a/test/presubmit-tests-sdk.sh b/test/presubmit-tests-sdk.sh
new file mode 100755
--- /dev/null
+++ b/test/presubmit-tests-sdk.sh
@@ -0,0 +1,41 @@
+#!/bin/bash -ex
+# Copyright 2020 Kubeflow Pipelines contributors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+source_root=$(pwd)
+
+python3 -m pip install --upgrade pip
+python3 -m pip install coverage==4.5.4 coveralls==1.9.2
+python3 -m pip install mock  # TODO: Remove when added to requirements
+python3 -m pip install -r "$source_root/sdk/python/requirements.txt"
+
+# Installing Argo CLI to lint all compiled workflows
+LOCAL_BIN="${HOME}/.local/bin"
+mkdir -p "$LOCAL_BIN"
+export PATH="${PATH}:$LOCAL_BIN" # Unnecessary - Travis already has it in PATH
+wget --quiet -O "${LOCAL_BIN}/argo" https://github.com/argoproj/argo/releases/download/v2.4.3/argo-linux-amd64 && chmod +x "${LOCAL_BIN}/argo"
+
+pushd "$source_root/sdk/python"
+python3 -m pip install -e .
+popd # Changing the current directory to the repo root for correct coverall paths
+coverage run --source=kfp --append -m unittest discover --verbose --start-dir sdk/python --pattern '*test*.py'
+
+set +x
+# export COVERALLS_REPO_TOKEN=$(gsutil cat gs://ml-pipeline-test-keys/coveralls_repo_token)
+set -x
+REPO_BASE="https://github.com/kubeflow/pipelines"
+export COVERALLS_SERVICE_NAME="prow"
+export COVERALLS_SERVICE_JOB_ID=$PROW_JOB_ID
+export CI_PULL_REQUEST="$REPO_BASE/pull/$PULL_NUMBER"
+# coveralls
diff --git a/test/presubmit-tests-tfx.sh b/test/presubmit-tests-tfx.sh
new file mode 100755
--- /dev/null
+++ b/test/presubmit-tests-tfx.sh
@@ -0,0 +1,62 @@
+#!/bin/bash -ex
+# Copyright 2020 Kubeflow Pipelines contributors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+source_root=$(pwd)
+
+python3 -m pip install --upgrade pip
+python3 -m pip install -r "$source_root/sdk/python/requirements.txt"
+# Additional dependencies
+#pip3 install coverage==4.5.4 coveralls==1.9.2 six>=1.13.0
+# Sample test infra dependencies
+pip3 install minio
+pip3 install junit_xml
+# Using Argo to lint all compiled workflows
+export LOCAL_BIN="${HOME}/.local/bin"
+mkdir -p "$LOCAL_BIN"
+export PATH="${PATH}:$LOCAL_BIN" # Unnecessary - Travis already has it in PATH
+wget --quiet -O "${LOCAL_BIN}/argo" https://github.com/argoproj/argo/releases/download/v2.4.3/argo-linux-amd64 && chmod +x "${LOCAL_BIN}/argo"
+
+pushd $source_root/sdk/python
+python3 -m pip install -e .
+popd # Changing the current directory to the repo root for correct coverall paths
+
+# Test against TFX
+# Compile and setup bazel for compiling the protos
+# Instruction from https://docs.bazel.build/versions/master/install-ubuntu.html
+curl -sSL https://github.com/bazelbuild/bazel/releases/download/3.4.1/bazel-3.4.1-installer-linux-x86_64.sh -o bazel_installer.sh
+chmod +x bazel_installer.sh
+./bazel_installer.sh
+
+# Install TFX from head
+cd $source_root
+git clone https://github.com/tensorflow/tfx.git
+cd $source_root/tfx
+#pip3 install --upgrade pip
+#pip3 install --upgrade 'numpy>=1.16,<1.17'
+#set -x
+#set -e
+#python3 setup.py bdist_wheel
+#WHEEL_PATH=$(find dist -name "tfx-*.whl")
+#python3 -m pip install "${WHEEL_PATH}" --upgrade
+python3 -m pip install . --upgrade
+#set +e
+#set +x
+
+# Three KFP-related unittests
+cd $source_root/tfx/tfx/orchestration/kubeflow
+python3 kubeflow_dag_runner_test.py
+cd $source_root/tfx/tfx/examples/chicago_taxi_pipeline
+python3 taxi_pipeline_kubeflow_gcp_test.py
+python3 taxi_pipeline_kubeflow_local_test.py