Repository: kubeflow/pipelines

Issue:
Enable local test runner for Kubeflow Pipelines
Today, there is no way to run a KFP pipeline on a local machine (outside of running a mini Kubernetes cluster locally). This makes testing pipelines for user difficult as they always have to go through the steps of uploading to a cluster to run even small tests on the correctness of the pipeline. 

To solve this problem, it would be nice if KFP offers a way to run pipelines locally. Such a mode does not have to have full fidelity with what's possible in a cluster run with Argo to begin with. A simple approach of using `docker run` to run each step sequentially, with a mounted local volume for passing parameters, will go a long way towards satisfying most local-run use-cases. 

As a suggestion, we can modify the pipeline decorator to enable this behaviour when the user adds a keyword like `test = True`, i.e.

```
@dsl.Pipeline(
   name = "..",
   test = True,
   ...
```




Discussion:
Issue-Label Bot is automatically applying the label `feature_request` to this issue, with a confidence of 0.98. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! 

 Links: [dashboard](https://mlbot.net/data/kubeflow/pipelines), [app homepage](https://github.com/apps/issue-label-bot) and [code](https://github.com/hamelsmu/MLapp) for this bot.
Some ideal requirements to consider: 
- The local executor should support all the same features as execution within a GKE cluster. 
- The local executor should not duplicate the code of the controllers used to execute the pipeline. 
- The local executor should execute the yaml, not Python, so that both yaml and Python can be tested. 
>The local executor should support all the same features as execution within a GKE cluster.

I guess only the proper local kubernetes cluster can satisfy this requirement. The test environment needs Minikube installed.
> As a suggestion, we can modify the pipeline decorator to enable this behaviour when the user adds a keyword like `test = True`, i.e.

I think it's better have a function to perform this instead of overloading the decorator:

```
kfp.run_pipeline_locally(my_pipeline, arguments={...})
```
If we want to differentiate, we can have
* run_pipeline_locally
* run_pipeline_using_docker
* run_pipeline_on_kubernetes
* run_pipeline_on_kfp
BTW, what about providing users with a VM image that has all the tools installed (minikube and etc.) to make local execution easy? 
Thank you so much for tracking this issue. 

Another thing to consider is that we may want to have a way to inject a stub to a component inside the pipeline, if a component is making call to foreign services such as Dataflow.
related: #1104 
Closing as infeasible/obsolete for now.
@Ark-kun not sure if you intentionally meant to re-open this issue? Are you working on local runner for KFP? 
> @Ark-kun not sure if you intentionally meant to re-open this issue? Are you working on local runner for KFP?

I think that your idea is still pretty valid and useful. It would be nice to have that feature for component testing and experimentation. I've recently started creating components in my free time and testing them was not that easy without having a Kubernetes cluster.
Our SDK already runs some tests in a local environment, but it would be less hacky to just use docker.
So I just wanted to keep this feature request in mind.
The priority of this is not P0 or P1 of course.

What do you think?
FYI, I'm thinking on bring a lightweight local runner in a new design. It may rely on TFX SDK's docker launcher. It's still early investigation phase, together with IR work.
FYI, for testing TFX has this https://github.com/tensorflow/tfx/pull/1986 WIP.

That said it's pretty different than local dev.
(in IR based impl, we may handle it with Docker runner)
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

/lifecycle frozen
we'd still want this open
Are we interested in adding instructions on how to deploy locally to Minikube based upon the local code for development @Bobgy? Will not be a big change but would require the user to build all the images and add an overlay that uses these images instead of the official released images. 
@NikeNano I think that's a different topic, this issue is about creating a local runner that helps experiment with KFP. Rather than developing KFP.

Resolving patch:
diff --git a/sdk/python/kfp/__init__.py b/sdk/python/kfp/__init__.py
--- a/sdk/python/kfp/__init__.py
+++ b/sdk/python/kfp/__init__.py
@@ -14,7 +14,7 @@
 
 # `kfp` is a namespace package.
 # https://packaging.python.org/guides/packaging-namespace-packages/#pkgutil-style-namespace-packages
-__path__ = __import__('pkgutil').extend_path(__path__, __name__)
+__path__ = __import__("pkgutil").extend_path(__path__, __name__)
 
 __version__ = '1.4.0-rc.1'
 
@@ -23,4 +23,5 @@
 from . import dsl
 from ._client import Client
 from ._config import *
+from ._local_client import LocalClient
 from ._runners import *
diff --git a/sdk/python/kfp/_local_client.py b/sdk/python/kfp/_local_client.py
new file mode 100644
--- /dev/null
+++ b/sdk/python/kfp/_local_client.py
@@ -0,0 +1,526 @@
+# Copyright 2021 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import datetime
+import json
+import logging
+import os
+import re
+import subprocess
+from collections import deque
+from typing import Any, Callable, Dict, List, Mapping, Union, cast
+
+from . import dsl
+from .compiler.compiler import sanitize_k8s_name
+
+
+class _Dag:
+    """DAG stands for Direct Acyclic Graph.
+
+    DAG here is used to decide the order to execute pipeline ops.
+
+    For more information on DAG, please refer to `wiki <https://en.wikipedia.org/wiki/Directed_acyclic_graph>`_.
+
+    """
+
+    def __init__(self, nodes: List[str]) -> None:
+        """
+
+        Args::
+          nodes: List of DAG nodes, each node is identified by an unique name.
+        """
+        self._graph = {node: [] for node in nodes}
+        self._reverse_graph = {node: [] for node in nodes}
+
+    @property
+    def graph(self):
+        return self._graph
+
+    @property
+    def reverse_graph(self):
+        return self._reverse_graph
+
+    def add_edge(self, edge_source: str, edge_target: str) -> None:
+        """Add an edge between DAG nodes.
+
+        Args::
+          edge_source: the source node of the edge
+          edge_target: the target node of the edge
+        """
+        self._graph[edge_source].append(edge_target)
+        self._reverse_graph[edge_target].append(edge_source)
+
+    def get_follows(self, source_node: str) -> List[str]:
+        """Get all target nodes start from the specified source node
+
+        Args::
+          source_node: the source node
+        """
+        return self._graph.get(source_node, [])
+
+    def get_dependencies(self, target_node: str) -> List[str]:
+        """Get all source nodes end with the specified target node
+
+        Args::
+          target_node: the target node
+        """
+        return self._reverse_graph.get(target_node, [])
+
+    def topological_sort(self) -> List[str]:
+        """ List DAG nodes in topological order. """
+
+        in_degree = {node: 0 for node in self._graph.keys()}
+
+        for i in self._graph:
+            for j in self._graph[i]:
+                in_degree[j] += 1
+
+        queue = deque()
+        for node, degree in in_degree.items():
+            if degree == 0:
+                queue.append(node)
+
+        sorted_nodes = []
+
+        while queue:
+            u = queue.popleft()
+            sorted_nodes.append(u)
+
+            for node in self._graph[u]:
+                in_degree[node] -= 1
+
+                if in_degree[node] == 0:
+                    queue.append(node)
+
+        return sorted_nodes
+
+
+def _extract_pipeline_param(param: str) -> dsl.PipelineParam:
+    """ Extract PipelineParam from string """
+    matches = re.findall(r"{{pipelineparam:op=([\w\s_-]*);name=([\w\s_-]+)}}", param)
+    op_dependency_name = matches[0][0]
+    output_file_name = matches[0][1]
+    return dsl.PipelineParam(output_file_name, op_dependency_name)
+
+
+def _get_op(ops: List[dsl.ContainerOp], op_name: str) -> Union[dsl.ContainerOp, None]:
+    """ Get the first op with specified op name """
+    return next(filter(lambda op: op.name == op_name, ops), None)
+
+
+def _get_subgroup(
+    groups: List[dsl.OpsGroup], group_name: str
+) -> Union[dsl.OpsGroup, None]:
+    """ Get the first OpsGroup with specified group name """
+    return next(filter(lambda g: g.name == group_name, groups), None)
+
+
+class LocalClient:
+    class ExecutionMode:
+        """Configuration to decide whether the client executes a component in docker or
+        in local process.
+        """
+
+        DOCKER = "docker"
+        LOCAL = "local"
+
+        def __init__(
+            self,
+            mode: str = DOCKER,
+            images_to_exclude: List[str] = [],
+            ops_to_exclude: List[str] = [],
+        ) -> None:
+            """Constructor
+
+            Args:
+                mode: Default execution mode, default 'docker'
+                images_to_exclude: If the image of op is in images_to_exclude, the op is
+                    executed in the mode different from default_mode.
+                ops_to_exclude: If the name of op is in ops_to_exclude, the op is
+                    executed in the mode different from default_mode.
+            """
+            if mode not in [self.DOCKER, self.LOCAL]:
+                raise Exception("Invalid execution mode, must be docker of local")
+            self._mode = mode
+            self._images_to_exclude = images_to_exclude
+            self._ops_to_exclude = ops_to_exclude
+
+        @property
+        def mode(self):
+            return self._mode
+
+        @property
+        def images_to_exclude(self):
+            return self._images_to_exclude
+
+        @property
+        def ops_to_exclude(self):
+            return self._ops_to_exclude
+
+    def __init__(self, pipeline_root: str = "/tmp") -> None:
+        """Construct the instance of LocalClient
+
+        Args：
+            pipeline_root: The root directory where the output artifact of component
+              will be savad.
+        """
+        self._pipeline_root = pipeline_root
+
+    def _find_base_group(
+        self, groups: List[dsl.OpsGroup], op_name: str
+    ) -> Union[dsl.OpsGroup, None]:
+        """ Find the base group of op in candidate group list. """
+        if groups is None or len(groups) == 0:
+            return None
+        for group in groups:
+            if _get_op(group.ops, op_name):
+                return group
+            else:
+                _parent_group = self._find_base_group(group.groups, op_name)
+                if _parent_group:
+                    return group
+
+        return None
+
+    def _create_group_dag(self, pipeline_dag: _Dag, group: dsl.OpsGroup) -> _Dag:
+        """Create DAG within current group, it's a DAG of direct ops and direct subgroups.
+
+        Each node of the DAG is either an op or a subgroup.
+        For each node in current group, if one of its DAG follows is also an op in
+        current group, add an edge to this follow op, otherwise, if this follow belongs
+        to subgroups, add an edge to its subgroup. If this node has dependency from
+        subgroups, then add an edge from this subgroup to current node.
+        """
+        group_dag = _Dag([op.name for op in group.ops] + [g.name for g in group.groups])
+
+        for op in group.ops:
+            for follow in pipeline_dag.get_follows(op.name):
+                if _get_op(group.ops, follow) is not None:
+                    # add edge between direct ops
+                    group_dag.add_edge(op.name, follow)
+                else:
+                    _base_group = self._find_base_group(group.groups, follow)
+                    if _base_group:
+                        # add edge to direct subgroup
+                        group_dag.add_edge(op.name, _base_group.name)
+
+            for dependency in pipeline_dag.get_dependencies(op.name):
+                if _get_op(group.ops, dependency) is None:
+                    _base_group = self._find_base_group(group.groups, dependency)
+                    if _base_group:
+                        # add edge from direct subgroup
+                        group_dag.add_edge(_base_group.name, op.name)
+
+        return group_dag
+
+    def _create_op_dag(self, p: dsl.Pipeline) -> _Dag:
+        """ Create the DAG of the pipeline ops. """
+        dag = _Dag(p.ops.keys())
+
+        for op in p.ops.values():
+            # dependencies defined by inputs
+            for input_value in op.inputs:
+                if isinstance(input_value, dsl.PipelineParam):
+                    input_param = _extract_pipeline_param(input_value.pattern)
+                    if input_param.op_name:
+                        dag.add_edge(input_param.op_name, op.name)
+                    else:
+                        logging.debug("%s depend on pipeline param", op.name)
+
+            # explicit dependencies of current op
+            for dependent in op.dependent_names:
+                dag.add_edge(dependent, op.name)
+        return dag
+
+    def _make_output_file_path_unique(
+        self, run_name: str, op_name: str, output_file: str
+    ) -> str:
+        """Alter the file path of output artifact to make sure it's unique in local runner.
+
+        kfp compiler will bound a tmp file for each component output, which is unique
+        in kfp runtime, but not unique in local runner. We alter the file path of the
+        name of current run and op, to make it unique in local runner.
+        """
+        return re.sub(
+            "/tmp",
+            "{pipeline_root}/{run_name}/{op_name}".format(
+                pipeline_root=self._pipeline_root,
+                run_name=run_name,
+                op_name=op_name.lower(),
+            ),
+            output_file,
+        )
+
+    def _get_output_file_path(
+        self,
+        run_name: str,
+        pipeline: dsl.Pipeline,
+        op_name: str,
+        output_name: str = None,
+    ) -> str:
+        """ Get the file path of component output. """
+
+        op_dependency = pipeline.ops[op_name]
+        if output_name is None and len(op_dependency.file_outputs) == 1:
+            output_name = next(iter(op_dependency.file_outputs.keys()))
+        output_file = op_dependency.file_outputs[output_name]
+        unique_output_file = self._make_output_file_path_unique(
+            run_name, op_name, output_file
+        )
+        return unique_output_file
+
+    def _generate_cmd_for_subprocess_execution(
+        self,
+        run_name: str,
+        pipeline: dsl.Pipeline,
+        op: dsl.ContainerOp,
+        stack: Dict[str, Any],
+    ) -> List[str]:
+        """ Generate shell command to run the op locally. """
+        cmd = op.command + op.arguments
+
+        # In debug mode, for `python -c cmd` format command, pydev will insert code before
+        # `cmd`, but there is no newline at the end of the inserted code, which will cause
+        # syntax error, so we add newline before `cmd`.
+        for i in range(len(cmd)):
+            if cmd[i] == "-c":
+                cmd[i + 1] = "\n" + cmd[i + 1]
+
+        for index, cmd_item in enumerate(cmd):
+            if cmd_item in stack:  # Argument is LoopArguments item
+                cmd[index] = str(stack[cmd_item])
+            elif cmd_item in op.file_outputs.values():  # Argument is output file
+                output_name = next(
+                    filter(lambda item: item[1] == cmd_item, op.file_outputs.items())
+                )[0]
+                output_param = op.outputs[output_name]
+                output_file = cmd_item
+                output_file = self._make_output_file_path_unique(
+                    run_name, output_param.op_name, output_file
+                )
+
+                os.makedirs(os.path.dirname(output_file), exist_ok=True)
+                cmd[index] = output_file
+            elif (
+                cmd_item in op.input_artifact_paths.values()
+            ):  # Argument is input artifact file
+                input_name = next(
+                    filter(
+                        lambda item: item[1] == cmd_item,
+                        op.input_artifact_paths.items(),
+                    )
+                )[0]
+                input_param_pattern = op.artifact_arguments[input_name]
+                pipeline_param = _extract_pipeline_param(input_param_pattern)
+                input_file = self._get_output_file_path(
+                    run_name, pipeline, pipeline_param.op_name, pipeline_param.name
+                )
+
+                cmd[index] = input_file
+
+        return cmd
+
+    def _generate_cmd_for_docker_execution(
+        self,
+        run_name: str,
+        pipeline: dsl.Pipeline,
+        op: dsl.ContainerOp,
+        stack: Dict[str, Any],
+    ) -> List[str]:
+        """ Generate the command to run the op in docker locally. """
+        cmd = self._generate_cmd_for_subprocess_execution(run_name, pipeline, op, stack)
+
+        docker_cmd = [
+            "docker",
+            "run",
+            "-v",
+            "{pipeline_root}:{pipeline_root}".format(pipeline_root=self._pipeline_root),
+            op.image,
+        ] + cmd
+        return docker_cmd
+
+    def _run_group_dag(
+        self,
+        run_name: str,
+        pipeline: dsl.Pipeline,
+        pipeline_dag: _Dag,
+        current_group: dsl.OpsGroup,
+        stack: Dict[str, Any],
+        execution_mode: ExecutionMode,
+    ):
+        """Run ops in current group in topological order
+
+        Args:
+            pipeline: kfp.dsl.Pipeline
+            pipeline_dag: DAG of pipeline ops
+            current_group: current ops group
+            stack: stack to trace `LoopArguments`
+            execution_mode: Configuration to decide whether the client executes
+                component in docker or in local process.
+        """
+        group_dag = self._create_group_dag(pipeline_dag, current_group)
+
+        for node in group_dag.topological_sort():
+            subgroup = _get_subgroup(current_group.groups, node)
+            if subgroup is not None:  # Node of DAG is subgroup
+                self._run_group(
+                    run_name, pipeline, pipeline_dag, subgroup, stack, execution_mode
+                )
+            else:  # Node of DAG is op
+                op = _get_op(current_group.ops, node)
+
+                execution_mode = (
+                    execution_mode if execution_mode else LocalClient.ExecutionMode()
+                )
+                can_run_locally = execution_mode.mode == LocalClient.ExecutionMode.LOCAL
+                exclude = (
+                    op.image in execution_mode.images_to_exclude
+                    or op.name in execution_mode.ops_to_exclude
+                )
+                if exclude:
+                    can_run_locally = not can_run_locally
+
+                if can_run_locally:
+                    cmd = self._generate_cmd_for_subprocess_execution(
+                        run_name, pipeline, op, stack
+                    )
+                else:
+                    cmd = self._generate_cmd_for_docker_execution(
+                        run_name, pipeline, op, stack
+                    )
+                process = subprocess.Popen(
+                    subprocess.list2cmdline(cmd) if "sh" != cmd[0] else cmd,
+                    shell="sh" != cmd[0],
+                    stdout=subprocess.PIPE,
+                    stderr=subprocess.PIPE,
+                    universal_newlines=True,
+                )
+                # TODO support async process
+                logging.info("start task：%s", op.name)
+                stdout, stderr = process.communicate()
+                if stdout:
+                    logging.info(stdout)
+                if stderr:
+                    logging.error(stderr)
+                if process.returncode != 0:
+                    logging.error(cmd)
+                    break
+
+    def _run_group(
+        self,
+        run_name: str,
+        pipeline: dsl.Pipeline,
+        pipeline_dag: _Dag,
+        current_group: dsl.OpsGroup,
+        stack: Dict[str, Any],
+        execution_mode: ExecutionMode,
+    ):
+        """Run all ops in current group
+
+        Args:
+            run_name: str, the name of this run, can be used to query the run result
+            pipeline: kfp.dsl.Pipeline
+            pipeline_dag: DAG of pipeline ops
+            current_group: current ops group
+            stack: stack to trace `LoopArguments`
+            execution_mode: Configuration to decide whether the client executes
+                component in docker or in local process.
+        """
+        if current_group.type == dsl.ParallelFor.TYPE_NAME:
+            current_group = cast(dsl.ParallelFor, current_group)
+
+            if current_group.items_is_pipeline_param:
+                _loop_args = current_group.loop_args
+                _param_name = _loop_args.name[
+                    : -len(_loop_args.LOOP_ITEM_NAME_BASE) - 1
+                ]
+
+                _op_dependency = pipeline.ops[_loop_args.op_name]
+                _list_file = _op_dependency.file_outputs[_param_name]
+                _altered_list_file = self._make_output_file_path_unique(
+                    run_name, _loop_args.op_name, _list_file
+                )
+                with open(_altered_list_file, "r") as f:
+                    _param_values = json.load(f)
+                for index, _param_value in enumerate(_param_values):
+                    if isinstance(_param_values, object):
+                        _param_value = json.dumps(_param_value)
+                    stack[_loop_args.pattern] = _param_value
+                    loop_run_name = "{run_name}/{loop_index}".format(
+                        run_name=run_name, loop_index=index
+                    )
+                    self._run_group_dag(
+                        loop_run_name,
+                        pipeline,
+                        pipeline_dag,
+                        current_group,
+                        stack,
+                        execution_mode,
+                    )
+                    del stack[_loop_args.pattern]
+            else:
+                raise Exception("Not implemented")
+        else:
+            self._run_group_dag(
+                run_name, pipeline, pipeline_dag, current_group, stack, execution_mode
+            )
+
+    def create_run_from_pipeline_func(
+        self,
+        pipeline_func: Callable,
+        arguments: Mapping[str, str],
+        execution_mode: ExecutionMode = ExecutionMode(),
+    ):
+        """Runs a pipeline locally, either using Docker or in a local process.
+
+        Parameters:
+          pipeline_func: pipeline function
+          arguments: Arguments to the pipeline function provided as a dict, reference
+              to `kfp.client.create_run_from_pipeline_func`
+          execution_mode: Configuration to decide whether the client executes component
+              in docker or in local process.
+        """
+
+        class RunPipelineResult:
+            def __init__(
+                self, client: LocalClient, pipeline: dsl.Pipeline, run_id: str
+            ):
+                self._client = client
+                self._pipeline = pipeline
+                self.run_id = run_id
+
+            def get_output_file(self, op_name: str, output: str = None):
+                return self._client._get_output_file_path(
+                    self.run_id, self._pipeline, op_name, output
+                )
+
+            def __repr__(self):
+                return "RunPipelineResult(run_id={})".format(self.run_id)
+
+        pipeline_name = sanitize_k8s_name(
+            getattr(pipeline_func, "_component_human_name", None)
+            or pipeline_func.__name__
+        )
+        with dsl.Pipeline(pipeline_name) as pipeline:
+            pipeline_func(**arguments)
+
+        run_version = datetime.datetime.now().strftime("%Y%m%d%H%M%S")
+        run_name = pipeline.name.replace(" ", "_").lower() + "_" + run_version
+
+        pipeline_dag = self._create_op_dag(pipeline)
+        self._run_group(
+            run_name, pipeline, pipeline_dag, pipeline.groups[0], {}, execution_mode
+        )
+
+        return RunPipelineResult(self, pipeline, run_name)
diff --git a/sdk/python/kfp/_runners.py b/sdk/python/kfp/_runners.py
--- a/sdk/python/kfp/_runners.py
+++ b/sdk/python/kfp/_runners.py
@@ -13,24 +13,25 @@
 # limitations under the License.
 
 __all__ = [
-    'run_pipeline_func_on_cluster',
+    "run_pipeline_func_on_cluster",
+    "run_pipeline_func_locally",
 ]
 
 
-from typing import Mapping, Callable
+from typing import Callable, List, Mapping
 
-from . import Client
-from . import dsl
+from . import Client, LocalClient, dsl
 
 
 def run_pipeline_func_on_cluster(
     pipeline_func: Callable,
     arguments: Mapping[str, str],
-    run_name : str = None,
-    experiment_name : str = None,
-    kfp_client : Client = None,
-    pipeline_conf: dsl.PipelineConf = None):
-    '''Runs pipeline on KFP-enabled Kubernetes cluster.
+    run_name: str = None,
+    experiment_name: str = None,
+    kfp_client: Client = None,
+    pipeline_conf: dsl.PipelineConf = None,
+):
+    """Runs pipeline on KFP-enabled Kubernetes cluster.
 
     This command compiles the pipeline function, creates or gets an experiment
     and submits the pipeline for execution.
@@ -49,6 +50,45 @@ def run_pipeline_func_on_cluster(
       pipeline_conf: Optional. kfp.dsl.PipelineConf instance. Can specify op
         transforms, image pull secrets and other pipeline-level configuration
         options.
-    '''
+    """
     kfp_client = kfp_client or Client()
-    return kfp_client.create_run_from_pipeline_func(pipeline_func, arguments, run_name, experiment_name, pipeline_conf)
+    return kfp_client.create_run_from_pipeline_func(
+        pipeline_func, arguments, run_name, experiment_name, pipeline_conf
+    )
+
+
+def run_pipeline_func_locally(
+    pipeline_func: Callable,
+    arguments: Mapping[str, str],
+    local_client: LocalClient = None,
+    execution_mode: LocalClient.ExecutionMode = LocalClient.ExecutionMode(),
+):
+    """Runs a pipeline locally, either using Docker or in a local process.
+
+    Feature stage:
+    [Alpha](https://github.com/kubeflow/pipelines/blob/master/docs/release/feature-stages.md#alpha)
+
+    In this alpha implementation, we support:
+      * Control flow: Condition, ParallelFor
+      * Data passing: InputValue, InputPath, OutputPath
+
+    And we don't support:
+      * Control flow: ExitHandler, Graph, SubGraph
+      * ContainerOp with environment variables, init containers, sidecars, pvolumes
+      * ResourceOp
+      * VolumeOp
+      * Caching
+
+    Args:
+      pipeline_func: A function that describes a pipeline by calling components
+        and composing them into execution graph.
+      arguments: Arguments to the pipeline function provided as a dict.
+        reference to `kfp.client.create_run_from_pipeline_func`
+      local_client: Optional. An instance of kfp.LocalClient
+      execution_mode: Configuration to decide whether the client executes component
+        in docker or in local process.
+    """
+    local_client = local_client or LocalClient()
+    return local_client.create_run_from_pipeline_func(
+        pipeline_func, arguments, execution_mode=execution_mode
+    )


Test patch:
diff --git a/sdk/python/tests/local_runner_test.py b/sdk/python/tests/local_runner_test.py
new file mode 100644
--- /dev/null
+++ b/sdk/python/tests/local_runner_test.py
@@ -0,0 +1,213 @@
+# Copyright 2021 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest
+from typing import Callable
+
+import kfp
+from kfp import LocalClient, run_pipeline_func_locally
+
+InputPath = kfp.components.InputPath()
+OutputPath = kfp.components.OutputPath()
+
+BASE_IMAGE = "python:3.7"
+
+
+def light_component(
+    base_image: str = BASE_IMAGE,
+):
+    """Decorator of kfp light component with customized parameters
+
+    Usage:
+    ```python
+    @light_component(base_image="python:3.7")
+    def a_component(src: kfp.components.InputPath(), ...):
+        ...
+    ```
+    """
+
+    def wrapper(func: Callable):
+        return kfp.components.create_component_from_func(
+            func=func,
+            base_image=base_image,
+        )
+
+    return wrapper
+
+
+@light_component()
+def hello(name: str):
+    print(f"hello {name}")
+
+
+@light_component()
+def local_loader(src: str, dst: kfp.components.OutputPath()):
+    import os
+    import shutil
+
+    if os.path.exists(src):
+        shutil.copyfile(src, dst)
+
+
+@light_component()
+def flip_coin(dst: kfp.components.OutputPath()):
+    import random
+
+    result = "head" if random.randint(0, 1) == 0 else "tail"
+    with open(dst, "w") as f:
+        f.write(result)
+
+
+@light_component()
+def list(dst: kfp.components.OutputPath()):
+    import json
+
+    with open(dst, "w") as f:
+        json.dump(["hello", "world", "kfp"], f)
+
+
+@light_component()
+def component_connect_demo(
+    src: kfp.components.InputPath(), dst: kfp.components.OutputPath()
+):
+    with open(src, "r") as f:
+        line = f.readline()
+        print(f"read first line: {line}")
+        with open(dst, "w") as fw:
+            fw.write(f"{line} copied")
+
+
+class LocalRunnerTest(unittest.TestCase):
+    def setUp(self):
+        import tempfile
+
+        self.temp_file = tempfile.NamedTemporaryFile()
+        with open(self.temp_file.name, "w") as f:
+            f.write("hello world")
+
+    def test_run_local(self):
+        def _pipeline(name: str):
+            hello(name)
+
+        run_pipeline_func_locally(
+            _pipeline,
+            {"name": "world"},
+            execution_mode=LocalClient.ExecutionMode("local"),
+        )
+
+    def test_local_file(self):
+        def _pipeline(file_path: str):
+            local_loader(file_path)
+
+        run_result = run_pipeline_func_locally(
+            _pipeline,
+            {"file_path": self.temp_file.name},
+            execution_mode=LocalClient.ExecutionMode("local"),
+        )
+        output_file_path = run_result.get_output_file("local-loader")
+
+        with open(output_file_path, "r") as f:
+            line = f.readline()
+            assert "hello" in line
+
+    def test_condition(self):
+        def _pipeline():
+            _flip = flip_coin()
+            with kfp.dsl.Condition(_flip.output == "head"):
+                hello("head")
+
+            with kfp.dsl.Condition(_flip.output == "tail"):
+                hello("tail")
+
+        run_pipeline_func_locally(
+            _pipeline, {}, execution_mode=LocalClient.ExecutionMode("local")
+        )
+
+    def test_for(self):
+        @light_component()
+        def cat(item, dst: OutputPath):
+            with open(dst, "w") as f:
+                f.write(item)
+
+        def _pipeline():
+            with kfp.dsl.ParallelFor(list().output) as item:
+                cat(item)
+
+        run_pipeline_func_locally(
+            _pipeline, {}, execution_mode=LocalClient.ExecutionMode("local")
+        )
+
+    def test_connect(self):
+        def _pipeline():
+            _local_loader = local_loader(self.temp_file.name)
+            component_connect_demo(_local_loader.output)
+
+        run_result = run_pipeline_func_locally(
+            _pipeline, {}, execution_mode=LocalClient.ExecutionMode("local")
+        )
+        output_file_path = run_result.get_output_file("component-connect-demo")
+
+        with open(output_file_path, "r") as f:
+            line = f.readline()
+            assert "copied" in line
+
+    def test_command_argument_in_any_format(self):
+        def echo():
+            return kfp.dsl.ContainerOp(
+                name="echo",
+                image=BASE_IMAGE,
+                command=["echo", "hello world", ">", "/tmp/outputs/output_file"],
+                arguments=[],
+                file_outputs={"output": "/tmp/outputs/output_file"},
+            )
+
+        def _pipeline():
+            _echo = echo()
+            component_connect_demo(_echo.output)
+
+        run_pipeline_func_locally(
+            _pipeline, {}, execution_mode=LocalClient.ExecutionMode("local")
+        )
+
+    def test_execution_mode_exclude_op(self):
+        @light_component(base_image="image_not_exist")
+        def cat_on_image_not_exist(name: str, dst: OutputPath):
+            with open(dst, "w") as f:
+                f.write(name)
+
+        def _pipeline():
+            cat_on_image_not_exist("exclude ops")
+
+        run_result = run_pipeline_func_locally(
+            _pipeline,
+            {},
+            execution_mode=LocalClient.ExecutionMode(mode="docker"),
+        )
+        output_file_path = run_result.get_output_file("cat-on-image-not-exist")
+        import os
+
+        assert not os.path.exists(output_file_path)
+
+        run_result = run_pipeline_func_locally(
+            _pipeline,
+            {},
+            execution_mode=LocalClient.ExecutionMode(
+                mode="docker", ops_to_exclude=["cat-on-image-not-exist"]
+            ),
+        )
+        output_file_path = run_result.get_output_file("cat-on-image-not-exist")
+
+        with open(output_file_path, "r") as f:
+            line = f.readline()
+            assert "exclude ops" in line
diff --git a/test/presubmit-tests-sdk.sh b/test/presubmit-tests-sdk.sh
--- a/test/presubmit-tests-sdk.sh
+++ b/test/presubmit-tests-sdk.sh
@@ -17,7 +17,7 @@ source_root=$(pwd)
 
 python3 -m pip install --upgrade pip
 python3 -m pip install coverage==4.5.4 coveralls==1.9.2
-python3 -m pip install mock  # TODO: Remove when added to requirements
+python3 -m pip install mock # TODO: Remove when added to requirements
 python3 -m pip install -r "$source_root/sdk/python/requirements.txt"
 python3 -m pip install --upgrade protobuf