Package: ibm-watson-pipelines
IBM Watson Pipelines Python ClientThis package provides various utilities for working with IBM Watson Pipelines.
Its primary usage is to enable users to store artifact results of a notebook run.UsageConstructionWatsonPipelinesclient is constructed from IAM APIKEY, which can be provided
in a few ways:explicitly:fromibm_watson_pipelinesimportWatsonPipelinesclient=WatsonPipelines(apikey)# orclient=WatsonPipelines.from_apikey(apikey)# orclient=WatsonPipelines.from_token(token)implicitly:APIKEY=...exportAPIKEYorUSER_ACCESS_TOKEN=...exportUSER_ACCESS_TOKENfromibm_watson_pipelinesimportWatsonPipelines# use APIKEYclient=WatsonPipelines.from_apikey()# use USER_ACCESS_TOKENclient=WatsonPipelines.from_token()# try APIKEY, if absent then USER_ACCESS_TOKEN:client=WatsonPipelines()# orclient=WatsonPipelines.new_instance()All of the above may also defineservice_nameandurl.The exact procedure of deciding which authentication method to use:Iffrom_apikeyorfrom_tokenis used, the method is forced.If constructor is used but eitherapikeyorbearer_tokenargument
was provided, that method will be forced (if both are present,
an overloading error will be raised). Note that providing a nameless
argument is equivalent to providingapikey.If constructor ornew_instanceis used,APIKEYenv-var is used.If constructor ornew_instanceis used, butAPIKEYenv-var is not
present,USER_ACCESS_TOKENenv-var is used.If none of the above matches, an error is returned.Usage in Python notebooksNotebooks run in IBM Watson Pipelines get inputs and expose
outputs as a node:{
  "id": ...,
  "type": "execution_node",
  "op": "run_container",
  "app_data": {
    "pipeline_data": {
      "name": ...,
      "config": {
        "link": {
          "component_id_ref": "run-notebook"
        }
      },
      "inputs": [
        ...,
        {
          "name": "model_name",
          "group": "env_variables",
          "type": "String",
          "value_from": ...
        }
      ],
      "outputs": [
        {
          "name": "trained_model",
          "group": "output_variables",
          "type": {
            "CPDPath": {
              "path_type": "resource",
              "resource_type": "asset",
              "asset_type": "wml_model"
            }
          }
        }
      ]
    }
  },
  ...
}Inside of the notebook, inputs are available as environmental
variables:model_name=os.environ['model_name']Outputs are exposed using sdk method,store_results:client=WSPipelines.from_apikey(...)client.store_results({"trained_model":...//cpdpathtothetrainedmodel})Extracting credentialsOn public cloud, this client provides a method for easy retrieval of WML
instance credentials and scope storage credentials:client.get_wml_credentials()# the scope passed in notebook# orclient.get_wml_credentials("cpd:///projects/123456789")client.get_storage_credentials()# the scope passed in notebook# orclient.get_storage_credentials("cpd:///projects/123456789")Note how the result will vary depending on the authentication method
used to create the client.CPD-Path manipulationCPD-Path parsing is manipulation is also supported:fromibm_watson_pipelinesimportCpdScope,WatsonPipelinesclient=WatsonPipelines.from_apikey()scope=CpdScope.from_string("cpd:///projects/123456789")assertscope.scope_type()=="projects"assertscope.scope_id()=="123456789"client.get_wml_credentials(scope)Different kinds of CPD-Paths will have different properties, providing the same
interface across scopes, resource and file paths:fromibm_watson_pipelinesimportCpdPathscope_file_path=CpdPath.from_string("cpd:///projects/123456789/files/abc/def")assertscope_file_path.scope_type()=="projects"assertscope_file_path.scope_id()=="123456789"assertscope_file_path.file_path()=="/abc/def"connection_path=CpdPath.from_string("cpd:///projects/123456789/connections/3141592")assertconnection_path.scope_type()=="projects"assertconnection_path.scope_id()=="123456789"assertconnection_path.resource_type()=="connections"assertconnection_path.resource_id()=="3141592"connection_file_path=CpdPath.from_string("cpd:///projects/123456789/connections/3141592/files/~/abc/def")assertconnection_file_path.scope_type()=="projects"assertconnection_file_path.scope_id()=="123456789"assertconnection_file_path.resource_type()=="connections"assertconnection_file_path.resource_id()=="3141592"assertconnection_file_path.bucket_name()=="~"assertconnection_file_path.file_path()=="/abc/def"...additionally, for non-scope paths the scope can be extracted, if present:fromibm_watson_pipelinesimportCpdPathscope_path=CpdPath.from_string("cpd:///projects/123456789")connection_path=CpdPath.from_string("cpd:///projects/123456789/connections/3141592")assertconnection_path.scope()==scope_pathCustom components for use in the pipelineA custom pipeline component executes a script you write. You can use custom components to share reusable scripts between pipelines.You create custom components as project assets. You can then use the components in pipelines you create in that project. You can create as many custom components for pipelines as needed. Currently, to create a custom component you must create one programmatically, using a Python function.Creating a component as a project assetTo create a custom component, use the Python client to authenticate with IBM Watson Pipelines, code the component, then publish the component to the specified project. After it is available in the project, you can assign it to a node in a pipeline and run it as part of a pipeline flow.This example demonstrates the process of publishing a component that adds two numbers together.Publish a function as a component with the latest Python client. Run the following code in a Jupyter notebook in a project of your Cloud Pak for Data.# Install libraries!pipinstallibm-watson-pipelines==1.0.0# Authenticationfromibm_watson_pipelinesimportWatsonPipelinesapikey=''service_url='your_host_url'project_id='your_project_id'client=WatsonPipelines.from_apikey(apikey,url=service_url)# Define the function of the component# If you define the input parameters, users are required to# input them in the UIdefadd_two_numbers(a:int,b:int)->int:print('Adding numbers:{}+{}.'.format(a,b))returna+b+10# Other possible functions might be sending a Slack message,# or listing directories in a storage volume, and so on.# Publish the componentclient.publish_component(name='Add numbers',# Appears in UI as component namefunc=add_two_numbers,description='Custom component adding numbers',# Appears in UI as component descriptionproject_id=project_id,overwrite=True,# Overwrites an existing component with the same name)Manage pipeline componentslist components from a project:client.get_components(project_id=project_id)get a component by ID:client.get_component(project_id=project_id,component_id=component_id)get a component by name:client.get_component(project_id=project_id,name=component_name)publish a new component:client.publish_component(componentname)delete a component by ID:client.delete_component(project_id=project_id,component_id=component_id)