Package: mlpiper
READMEThe DataRobotmlpipermodule is designed to process and execute complex pipelines, that
consist of one or more components chained together such that output of a
previous component becomes the input to the next component. Each pipeline
has a particular purpose, such as to train a model or generate predictions.A single pipeline may include components from different languages, such as Python, R and Java.Installationpip install mlpiper[sagemaker][pyspark][wizard][mlops]Note:the extra installation options are as follows:sagemaker: provides support forsagemakerpipelines, which
requires proper AWS credentialspyspark:  provides support forpysparkpipelineswizard:   provides wizard capability to create a component
metadata file from a command lineQuickstartStepsCreate a pipeline. Open any text editor and copy the following pipeline description:{
      "name": "Simple MCenter runner test",
      "engineType": "Generic",
      "pipe": [
          {
              "name": "Source String",
              "id": 1,
              "type": "string-source",
              "parents": [],
              "arguments": {
                  "value": "Hello World: testing string source and sink"
              }
          },
          {
              "name": "Sink String",
              "id": 2,
              "type": "string-sink",
              "parents": [{"parent": 1, "output": 0}],
              "arguments": {
                  "expected-value": "Hello World: testing string source and sink"
              }
          }
      ]
  }Clonemlpiperrepohttps://github.com/mlpiper/mlpiper/Componentsstring-sourceandstring-sinkcan be found in the repo pathhttps://github.com/mlpiper/mlpiper/tree/master/reflex-algos/components/PythonOnce themlpiperpython package is installed, themlpipercommand line tool is available and can be used to execute the above pipeline and the components described in it. Run the example above with:mlpiper run -f ~/<pipeline description file> -r <path to mlpiper repo>/reflex-algos/components/PythonHow to construct a componentStepsCreate a directory, the name of which corresponds to the component's name (e.g., source_string)Create acomponent.jsonfile (JSON format) inside this directory and make sure to fill in all of the following fields:{
      "engineType": "Generic",
      "language": "Python",
      "userStandalone": false,
      "name": "<Component name (e.g., string_source)>",
      "label": "<A lable that is displayed in the UI>",
      "version": "<Component's version (e.g., 1.0.0)>",
      "group": "<One of the valid groups (e.g., "Connectors")>,
      "program": "<The Python component main script (e.g., string_source.py)>",
      "componentClass": "<The component class name (e.g., StringSource)
      "useMLStats": <true|false - (whether the components uses MLStats)>,
      "inputInfo": [
          {
           "description": "<Description>",
           "label": "<Lable name>",
           "defaultComponent": "",
           "type": "<A type used to verify matching connected legs>,
           "group": "<data|model|prediction|statistics|other>"
          },
          {...}
      ],
      "outputInfo": [
          <Same as inputInfo above>
      ],
      "arguments": [
          {
              "key": "<Unique argument key name>",
              "type": "int|long|float|str|bool",
              "label": "<A label that is displayed in the UI>",
              "description": "<Description>",
              "optional": <true|false>
          }
      ]
  }Create the main component script, which contains the component's class name.
This class should inherit from a 'Component' base class, which is taken frommlpiper.components.component. The class must implement thematerializefunction, with this prototype:def _materialize(self, parent_data_objs, user_data).
Here is a complete self contained example:from mlpiper.components import ConnectableComponent


  class StringSource(ConnectableComponent):
      def __init__(self, engine):
          super(self.__class__, self).__init__(engine)

      def _materialize(self, parent_data_objs, user_data):
          self._logger.info("Inside string source component")
          str_value = self._params.get('value', "default-string-value")

          return [str_value]Notes:A component can useself._loggerobject to print logs.A component may access to pipeline parameters viaself._paramsdictionary.The_materializefunction should return a list of objects or None otherwise.
This returned value will be used as an input for the next component
in the pipeline chain.Place the component's main program (*.py) inside a directory along with its JSON
description file and any other desired files.How to construct a pipelineStepsOpen any text editor and copy the following pipeline description:{
      "name": "Simple MCenter runner test",
      "engineType": "Generic",
      "pipe": [
          {
              "name": "Source String",
              "id": 1,
              "type": "string-source",
              "parents": [],
              "arguments": {
                  "value": "Hello World: testing string source and sink"
              }
          },
          {
              "name": "Sink String",
              "id": 2,
              "type": "string-sink",
              "parents": [{"parent": 1, "output": 0}],
              "arguments": {
                  "expected-value": "Hello World: testing string source and sink"
              }
          }
      ]
  }Notes:It is assumed that you've already constructed two components whose names
are:string-sourceandstring-sinkThe output of thestring-sourcecomponent (the value returned from_materializefunction) is supposed to become the input of thestring-sinkcomponent (an input to the_materializefunction)Save it with any desired nameHow to testOnce themlpiperpython package is installed,mlpipercommand line tool is available
and can be used to execute the above pipeline and the components described in it.There are three main commands that can be used as follows:deploy- Deploys a pipeline along with provided components into a given
directory. Once deployed, it can be executed directly from
the given directory.run- Deploys and then executes the pipeline.run-deployment- Executes an already-deployed pipeline.Examples:Prepare a deployment. The resulting directory will be copied to a docker container and run
there:mlpiper deploy -f p1.json -r ~/dev/components -d /tmp/ppDeploy & Run in-place:mlpiper run -f p1.json -r ~/dev/componentsDeploy & Run. Useful for development and debugging:mlpiper run -f p1.json -r ~/dev/components -d /tmp/ppRun a deployment. Usually non-interactive and called by another script:mlpiper run-deployment --deployment-dir /tmp/pp