Package: ml-comp
READMEThemlcompmodule is designed to process and execute complex pipelines, that consist of one or more components chained together such that output of a previous component becomes the input to the next component. Each pipeline has a particular purpose, such as to train a model or generate inferences.A single pipeline may include components from different languages, such as Python, R and Java.QuickstartStepsCreate a pipeline. Open any text editor and copy the following pipeline description:{
      "name": "Simple MCenter runner test",
      "engineType": "Generic",
      "pipe": [
          {
              "name": "Source String",
              "id": 1,
              "type": "string-source",
              "parents": [],
              "arguments": {
                  "value": "Hello World: testing string source and sink"
              }
          },
          {
              "name": "Sink String",
              "id": 2,
              "type": "string-sink",
              "parents": [{"parent": 1, "output": 0}],
              "arguments": {
                  "expected-value": "Hello World: testing string source and sink"
              }
          }
      ]
  }Clonemlpiperrepohttps://github.com/mlpiper/mlpiper/Componentsstring-sourceandstring-sinkcan be found in the repo pathhttps://github.com/mlpiper/mlpiper/tree/master/reflex-algos/components/PythonOnce theml-comppython package is installed, themlpipercommand line tool is available and can be used to execute the above pipeline and the components described in it. Run the example above with:mlpiper run -f ~/<pipeline description file> -r <path to mlpiper repo>/reflex-algos/components/Python/ -d <deployment dir>Use the--forceoption to overwrite the deployment directory.How to construct a componentStepsCreate a directory, the name of which corresponds to the component's name (e.g., source_string)Create acomponent.jsonfile (JSON format) inside this directory and make sure to fill in all of the following fields:{
      "engineType": "Generic",
      "language": "Python",
      "userStandalone": false,
      "name": "<Component name (e.g., string_source)>",
      "label": "<A lable that is displayed in the UI>",
      "version": "<Component's version (e.g., 1.0.0)>",
      "group": "<One of the valid groups (e.g., "Connectors")>,
      "program": "<The Python component main script (e.g., string_source.py)>",
      "componentClass": "<The component class name (e.g., StringSource)
      "useMLStats": <true|false - (whether the components uses MLStats)>,
      "inputInfo": [
          {
           "description": "<Description>",
           "label": "<Lable name>",
           "defaultComponent": "",
           "type": "<A type used to verify matching connected legs>,
           "group": "<data|model|prediction|statistics|other>"
          },
          {...}
      ],
      "outputInfo": [
          <Same as inputInfo above>
      ],
      "arguments": [
          {
              "key": "<Unique argument key name>",
              "type": "int|long|float|str|bool",
              "label": "<A label that is displayed in the UI>",
              "description": "<Description>",
              "optional": <true|false>
          }
      ]
  }Create the main component script, which contains the component's class name.
This class should inherit from a 'Component' base class, which is taken fromparallelm.components.component. The class must implement thematerializefunction, with this prototype:def _materialize(self, parent_data_objs, user_data).
Here is a complete self contained example:from parallelm.components import ConnectableComponent
  from parallelm.mlops import mlops


  class StringSource(ConnectableComponent):
      def __init__(self, engine):
          super(self.__class__, self).__init__(engine)

      def _materialize(self, parent_data_objs, user_data):
          self._logger.info("Inside string source component")
          str_value = self._params.get('value', "default-string-value")

          mlops.set_stat("Specific stat title", 1.0)
          mlops.set_stat("Specific stat title", 2.0)

          return [str_value]Notes:A component can useself._loggerobject to print logs.A component may access to pipeline parameters viaself._paramsdictionary.The_materializefunction should return a list of objects or None otherwise.
This returned value will be used as an input for the next component
in the pipeline chain.Place the component's main program (*.py) inside a directory along with its JSON
description file and any other desired files.How to construct a pipelineStepsOpen any text editor and copy the following pipeline description:{
      "name": "Simple MCenter runner test",
      "engineType": "Generic",
      "pipe": [
          {
              "name": "Source String",
              "id": 1,
              "type": "string-source",
              "parents": [],
              "arguments": {
                  "value": "Hello World: testing string source and sink"
              }
          },
          {
              "name": "Sink String",
              "id": 2,
              "type": "string-sink",
              "parents": [{"parent": 1, "output": 0}],
              "arguments": {
                  "expected-value": "Hello World: testing string source and sink"
              }
          }
      ]
  }Notes:It is assumed that you've already constructed two components whose names
are:string-sourceandstring-sinkThe output of thestring-sourcecomponent (the value returned from_materializefunction) is supposed to become the input of thestring-sinkcomponent (an input to the_materializefunction)Save it with any desired nameHow to testOnce theml-comppython package is installed,mlpipercommand line tool is available
and can be used to execute the above pipeline and the components described in it.There are three main commands that can be used as follows:deploy- Deploys a pipeline along with provided components into a given
directory. Once deployed, it can be executed directly from
the given directory.run- Deploys and then executes the pipeline.run-deployment- Executes an already-deployed pipeline.Examples:Prepare a deployment. The resulting directory will be copied to a docker container and run
there:mlpiper deploy -f p1.json -r ~/dev/components -d /tmp/ppDeploy & Run. Useful for development and debugging:mlpiper run -f p1.json -r ~/dev/components -d /tmp/ppUse--forceoption to overwrite deployment directoryRun a deployment. Usually non-interactive and called by another script:mlpiper run-deployment --deployment-dir /tmp/pp