ibm-research/WatsonxDocsQARetrieval huggingface dataset:
# Managing the lifecycle of notebooks and scripts #

After you have created and tested your notebooks, you can add them to pipelines, publish them to a catalog so that other catalog members can use the notebook in their projects, or share read\-only copies outside of Watson Studio so that people who aren't collaborators in your Watson Studio projects can see and use them\. R scripts and Shiny apps can't be published or shared using functionality in a project at this time\.

You can use any of these methods for notebooks:

<!-- <ul> -->

 *  [Add notebooks to a pipeline](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-orchestration-config.html)
 *  [Share a URL on social media](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/share-notebooks.html)
 *  [Publish on GitHub](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/github-integration.html)
 *  [Publish as a gist](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/create-gist.html)
 *  [Publish your notebook to a catalog](https://dataplatform.cloud.ibm.com/docs/content/wsj/catalog/publish-asset-project.html)

<!-- </ul> -->

Make sure that before you share or publish a notebook, you hide any sensitive code, like credentials, that you don't want others to see\! See [Hide sensitive cells in a notebook](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/hide_code.html)\.

**Parent topic:**[Notebooks and scripts](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/notebooks-and-scripts.html)

<!-- </article "role="article" "> -->
# Requirements for using custom components in ML models #

You can define your own transformers, estimators, functions, classes, and tensor operations in models that you deploy in IBM Watson Machine Learning as online deployments\.

## Defining and using custom components ##

To use custom components in your models, you need to package your custom components in a [Python distribution package](https://packaging.python.org/glossary/#term-distribution-package)\.

### Package requirements ###

<!-- <ul> -->

 *  The package type must be: [source distribution](https://packaging.python.org/glossary/#term-source-distribution-or-sdis) (distributions of type Wheel and Egg are not supported)
 *  The package file format must be: `.zip`
 *  Any third\-party dependencies for your custom components must be installable by `pip` and must be passed to the `install_requires` argument of the `setup` function of the `setuptools` library\.

<!-- </ul> -->

Refer to: [Creating a source distribution](https://docs.python.org/2/distutils/sourcedist.html)

### Storing your custom package ###

You must take extra steps when you store your trained model in the Watson Machine Learning repository:

<!-- <ul> -->

 *  Store your custom package in the [Watson Machine Learning repository](https://ibm.github.io/watson-machine-learning-sdk/core_api.html#ibm_watson_machine_learning.runtimes.Runtimes.store_library) (use the `runtimes.store_library` function from the Watson Machine Learning Python client, or the `store libraries` Watson Machine Learning CLI command\.)
 *  Create a runtime resource object that references your stored custom package, and then [store the runtime resource object](https://ibm.github.io/watson-machine-learning-sdk/core_api.html#ibm_watson_machine_learning.runtimes.Runtimes.store) in the Watson Machine Learning repository (use the `runtimes.store` function, or the `store runtimes` command\.)
 *  When you store your trained model in the Watson Machine Learning repository, reference your stored runtime resource in the [metadata](https://ibm.github.io/watson-machine-learning-sdk/core_api.html#client.Repository.store_model) that is passed to the `store_model` function (or the `store` command\.)

<!-- </ul> -->

## Supported frameworks ##

These frameworks support custom components:

<!-- <ul> -->

 *  Scikit\-learn
 *  XGBoost
 *  Tensorflow
 *  Python Functions
 *  Python Scripts
 *  Decision Optimization

<!-- </ul> -->

For more information, see [Supported frameworks](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/pm_service_supported_frameworks.html)

**Parent topic:**[Customizing deployment runtimes](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-customize.html)

<!-- </article "role="article" "> -->
# Customizing Watson Machine Learning deployment runtimes #

Create custom Watson Machine Learning deployment runtimes with libraries and packages that are required for your deployments\. You can build custom images based on deployment runtime images available in IBM Watson Machine Learning\. The images contain preselected open source libraries and selected IBM libraries\.

For a list of requirements for creating private Python packages, refer to [Requirements for using custom components in ML models](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-custom_libs_overview.html)\.

You can customize your deployment runtimes by [customizing Python runtimes with third\-party libraries and user\-created Python packages](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-create-custom-software-spec.html)

**Parent topic:**[Deploying and managing assets](https://dataplatform.cloud.ibm.com/docs/content/wsj/wmls/wmls-deploy-overview.html)

<!-- </article "role="article" "> -->
# Deleting a deployment #

Delete your deployment when you no longer need it to free up resources\. You can delete a deployment from a deployment space, or programmatically, by using the Python client or Watson Machine Learning APIs\.

## Deleting a deployment from a space ##

To remove a deployment:

<!-- <ol> -->

1.  Open the **Deployments** page of your deployment space\.
2.  Choose **Delete** from the action menu for the deployment name\.  
    ![Deleting a deployment](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/images/deploy-delete.png)

<!-- </ol> -->

## Deleting a deployment by using the Python client ##

Use the following method to delete the deployment\.

    client.deployments.delete(deployment_uid)

Returns a `SUCCESS` message\. To check that the deployment was removed, you can list deployments and make sure that the deleted deployment is no longer listed\.

    client.deployments.list()

Returns:

    ----  ----  -----  -------  -------------
    GUID  NAME  STATE  CREATED  ARTIFACT_TYPE
    ----  ----  -----  -------  -------------

## Deleting a deployment by using the REST API ##

Use the `DELETE` method for deleting a deployment\.

    DELETE /ml/v4/deployments/{deployment_id}

For more information, see [Delete](https://cloud.ibm.com/apidocs/machine-learning#deployments-delete)\.

For example, see the following code snippet:

    curl --location --request DELETE 'https://us-south.ml.cloud.ibm.com/ml/v4/deployments/:deployment_id?space_id=<string>&version=2020-09-01'

**Parent topic:**[Managing predictive deployments](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-deploy-general.html)

<!-- </article "role="article" "> -->
# Scaling a deployment #

When you create an online deployment for a model or function from a deployment space or programmatically, a single copy of the asset is deployed by default\. To increase scalability and availability, you can increase the number of copies (replicas) by editing the configuration of the deployment\. More copies allow for a larger volume of scoring requests\.

Deployments can be scaled in the following ways:

<!-- <ul> -->

 *  Update the configuration for a deployment in a deployment space\.
 *  Programmatically, using the Watson Machine Learning Python client library, or the Watson Machine Learning REST APIs\.

<!-- </ul> -->

## Changing the number of copies of an online deployment from a space ##

<!-- <ol> -->

1.  Click the **Deployment** tab of your deployment space\.
2.  From the action menu for your deployment name, click **Edit**\.
3.  In the **Edit deployment** dialog box, change the number of copies and click **Save**\.

<!-- </ol> -->

## Increasing the number of replicas of a deployment programmatically ##

To view or run a working sample of scaling a deployment programmatically, you can increase the number of replicas in the metadata for a deployment\.

### Python example ###

This example uses the Python client to set the number of replicas to 3\.

    change_meta = {
                    client.deployments.ConfigurationMetaNames.HARDWARE_SPEC: {
                                           "name":"S",
                                           "num_nodes":3}
                }
    
    client.deployments.update(<deployment_id>, change_meta)

The HARDWARE\_SPEC value includes a name because the API requires a name or an ID to be provided\.

### REST API example ###

    curl -k -X PATCH -d '[ { "op": "replace", "path": "/hardware_spec", "value": {  "name": "S", "num_nodes": 2  } } ]' <Deployment end-point URL>

You must specify a name for the `hardware_spec` value, but the argument is not applied for scaling\.

**Parent topic:**[Managing predictive deployments](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-deploy-general.html)

<!-- </article "role="article" "> -->
# Managing hardware configurations #

When you deploy certain assets in Watson Machine Learning, you can choose the type, size, and power of the hardware configuration that matches your computing needs\.

## Deployment types that require hardware specifications ##

Selecting a hardware specification is available for all [batch deployment](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/deploy-batch-details.html) types\. For [online deployments](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/deploy-online.html), you can select a specific hardware specification if you're deploying:

<!-- <ul> -->

 *  Python Functions
 *  Tensorflow models
 *  Models with custom software specifications

<!-- </ul> -->

## Hardware configurations available for deploying assets ##

<!-- <ul> -->

 *  `XS`: 1x4 = 1 vCPU and 4 GB RAM
 *  `S`: 2x8 = 2 vCPU and 8 GB RAM
 *  `M`: 4x16 = 4 vCPU and 16 GB RAM
 *  `L`: 8x32 = 8 vCPU and 32 GB RAM
 *  `XL`: 16x64 = 16 vCPU and 64 GB RAM

<!-- </ul> -->

You can use the `XS` configuration to deploy:

<!-- <ul> -->

 *  Python functions
 *  Python scripts
 *  R scripts
 *  Models based on custom libraries and custom images

<!-- </ul> -->

For Decision Optimization deployments, you can use these hardware specifications:

<!-- <ul> -->

 *  `S`
 *  `M`
 *  `L`
 *  `XL`

<!-- </ul> -->

## Learn more ##

<!-- <ul> -->

 *  [Monitoring account resource usage](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/monitor-resources.html)

<!-- </ul> -->

**Parent topic:**[Managing predictive deployments](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-deploy-general.html)

<!-- </article "role="article" "> -->
# Frameworks and software specifications in Watson Machine Learning #

You can use popular tools, libraries, and frameworks to train and deploy your machine learning models and functions\.

## Overview of software specifications ##

Software specifications define the programming language and version that you use for a building a model or a function\. You can use software specifications to configure the software that is used for running your models and functions\. You can also define the software version to be used and include your own extensions\. For example, you can use conda \.yml files or custom libraries\.

## Supported frameworks and software specifications ##

You can use predefined tools, libraries, and frameworks to train and deploy your machine learning models and functions\. Examples of supported frameworks include Scikit\-learn, Tensorflow, and more\.

For more information, see [Supported deployment frameworks and software specifications](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/pm_service_supported_frameworks.html)\.

![Frameworks and software specifications for model delpoyments](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/images/frameworks-software-specs.png)

## Managing outdated frameworks and software specifications ##

Update software specifications and frameworks in your models when they become outdated\. Sometimes, you can seamlessly update your assets\. In other cases, you must retrain or redeploy your assets\.

For more information, see [Managing outdated software specifications or frameworks](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-manage-outdated.html)\.

**Parent topic:**[Deploying assets with Watson Machine Learning](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-overview.html)

<!-- </article "role="article" "> -->
# Enabling model tracking with AI factsheets #

If your organization is using AI Factsheets as part of an AI governance strategy, you can track models after adding them to a space\.

Tracking a model populates a factsheet in an associated model use case\. The model use cases are maintained in a model inventory in a catalog, providing a way for all stakeholders to view the lifecyle details for a machine learning model\. From the inventory, collaborators can view the details for a model as it moves through the model lifecycle, including the request, development, deployment, and evaluation of the model\.

To enable model tracking by using AI Factsheets:

<!-- <ol> -->

1.  From the asset list in your space, click a model name and then click the **Model details** tab\.
2.  Click **Track this model**\.
3.  Associate the model with an existing model use case in the inventory or create a new use case\.
4.  Specify the details for the new use case, including specifying a catalog if you have access to more than one, and save to register the model\. A link to the model inventory is added to the model details page\.
5.  Click the link to open the model use case in the inventory\.
6.  Optional: update the model use case\. For example, add tags, supporting documentation, or other details\.

<!-- </ol> -->

<!-- </article "role="article" "> -->
# IBM Watson Pipelines #

The Watson Pipelines editor provides a graphical interface for orchestrating an end\-to\-end flow of assets from creation through deployment\. Assemble and configure a pipeline to create, train, deploy, and update machine learning models and Python scripts\.

To design a pipeline that you drag nodes onto the canvas, specify objects and parameters, then run and monitor the pipeline\.

## Automating the path to production ##

Putting a model into a product is a multi\-step process\. Data must be loaded and processed, models must be trained and tuned before they are deployed and tested\. Machine learning models require more observation, evaluation, and updating over time to avoid bias or drift\.

![Automating the AI lifecycle](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/images/oflow-cycle-3.svg)

Automating the pipeline makes it simpler to build, run, and evaluate a model in a cohesive way, to shorten the time from conception to production\. You can assemble the pipeline, then rapidly update and test modifications\. The Pipelines canvas provides tools to visualize the pipeline, customize it at run time with pipeline parameter variables, and then run it as a trial job or on a schedule\.

The Pipelines editor also allows for more cohesive collaboration between a data scientist and a ModelOps engineer\. A data scientist can create and train a model\. A ModelOps engineer can then automate the process of training, deploying, and evaluating the model after it is published to a production environment\.

## Next steps ##

[Add a pipeline](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-orchestration-get-started.html) to your project and get to know the canvas tools\.

## Additional resources ##

For more information, see this blog post about [automating the AI lifecycle with a pipeline flow](https://yairschiff.medium.com/automating-the-ai-lifecycle-with-ibm-watson-studio-orchestration-flow-4450f1d725d6)\.

<!-- </article "role="article" "> -->
## Accessing the components in your pipeline ##

When you use a pipeline to automate a flow, you must have access to all of the elements in the pipeline\. Make sure that you create and run pipelines with the proper access to all assets, projects, and spaces used in the pipeline\. Collaborators who run the pipeline must also be able to access the pipeline components\.

### Managing pipeline credentials ###

To run a job, the pipeline must have access to IBM Cloud credentials\. Typically, a pipeline uses your personal IBM Cloud API key to execute long\-running operations in the pipeline without disruption\. If credentials are not available when you create the job, you are prompted to supply an API key or create a new one\.

To generate an API key from your IBM Cloud user account, go to [Manage access and users \- API Keys](https://cloud.ibm.com/iam/apikeys) and create or select an API key for your user account\.

You can also generate and rotate API keys from **Profile and settings > User API key**\. For more information, see [Managing the user API key](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/admin-apikeys.html)\.

Alternatively, you can request that a key is generated for the pipeline\. In either scenario, name and copy the key, protecting it as you would a password\.

## Adding assets to a pipeline ##

When you create a pipeline, you add assets, such as data, notebooks, deployment jobs, or Data Refinery jobs to the pipeline to orchestrate a sequential process\. The ***strongly recommended*** method for adding assets to a pipeline is to collect the assets in the project containing the pipeline and use the asset browser to select project assets for the pipeline\.

Attention: Although you can include assets from other projects, doing so can introduce complexities and potential problems in your pipeline and could be prohibited in a future release\. The recommended practice is to use assets from the current project\.

**Parent topic:**[Getting started with Watson Pipelines](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-orchestration-get-started.html)

<!-- </article "role="article" "> -->
# Programming IBM Watson Pipelines #

You can program in a pipeline by using a notebook, or running Bash scripts in a pipeline\.

## Programming with Bash scripts ##

[Run Bash scripts](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-orchestration-config.html#run-bash) in a pipeline to compute or process data as part of the flow\.

## Programming with notebooks ##

You can use a notebook to run an end\-to\-end pipeline or to run parts of a pipeline, such as model training\.

<!-- <ul> -->

 *  For details on creating notebooks and for links to sample notebooks, see [Notebooks](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/notebook-editor.html)\.
 *  For details on running a notebook as a pipeline job, see [Run notebook job](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-orchestration-config.html#run-notebook)\.

<!-- </ul> -->

### Using the Python client ###

Use the [Watson Pipelines Python client](https://pypi.org/project/ibm-watson-pipelines/) for working with pipelines in a notebook\.

To install the library, use `pip` to install the latest package of `ibm-watson-pipelines` in your coding environment\. For example, run the following code in your notebook environment or console\.

    !pip install ibm-watson-pipelines

Use the client documentation for syntax and descriptions for commands that access pipeline components\.

### Go further ###

To learn more about how to orchestrate external tasks efficiently, see [Making tasks more efficiently with Tekton](https://medium.com/@rafal.bigaj/tekton-and-friends-how-to-orchestrate-external-tasks-efficiently-3fcacf882f6d), a key continuous delivery framework used for Pipelines\.

**Parent topic:**[Creating a pipeline](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-orchestration-create.html)

<!-- </article "role="article" "> -->
# Storage and data access for IBM Watson Pipelines #

Learn where files and data are stored outside of IBM Watson Pipelines and use it in a Pipelines\.

## Access data on Cloud Object Storage ##

File storage refers to the repository where you store assets to use with the pipeline\. It is a Cloud Object Storage bucket that is used as storage for a particular scope, such as a project or deployment space\.

A storage location is referenced by a Cloud Object Storage data connection in its scope\. Refer to a file by pointing to a location such as an object key in a dedicated, self\-managed bucket\.

**Parent topic:**[Creating a pipeline](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-orchestration-create.html)

<!-- </article "role="article" "> -->
# Deploying models with Watson Machine Learning #

Using IBM Watson Machine Learning, you can deploy models, scripts, and functions, manage your deployments, and prepare your assets to put into production to generate predictions and insights\.

This graphic illustrates a typical process for a machine learning model\. After you build and train a machine learning model, use Watson Machine Learning to deploy the model, manage the input data, and put your machine learning assets to use\.

![Building a machine learning model](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/images/ml_overview.svg)

## IBM Watson Machine Learning architecture and services ##

Watson Machine Learning is a service on IBM Cloud with features for training and deploying machine learning models and neural networks\. Built on a scalable, open source platform based on Kubernetes and Docker components, Watson Machine Learning enables you to build, train, deploy, and manage machine learning and deep learning models\.

## Deploying and managing models with Watson Machine Learning ##

Watson Machine Learning supports popular frameworks, including: TensorFlow, Scikit\-Learn, and PyTorch to build and deploy models\. For a list of supported frameworks, refer to [Supported frameworks](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/pm_service_supported_frameworks.html)\.

To build and train a model:

<!-- <ul> -->

 *  Use one of the tools that are listed in [Analyzing data and building models](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/data-science.html)\.
 *  [Import a model](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-importing-model.html) that you built and trained outside of Watson Studio\.

<!-- </ul> -->

### Deployment infrastructure ###

<!-- <ul> -->

 *  [Deploy trained models](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-deploy-general.html) as a web service or for batch processing\.
 *  [Deploy Python functions](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-deploy-py-function.html) to simplify AI solutions\.

<!-- </ul> -->

### Programming Interfaces ###

<!-- <ul> -->

 *  Use [Python client library](https://ibm.github.io/watson-machine-learning-sdk/) to work with all of your Watson Machine Learning assets in a notebook\.
 *  Use [REST API](https://cloud.ibm.com/apidocs/machine-learning) to call methods from the base URLs for the Watson Machine Learning API endpoints\.
 *  When you call the API, use the URL and add the path for each method to form the complete API endpoint for your requests\. For details on checking endpoints, refer to [Looking up a deployment endpoint](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/deploy-online.html)\.

<!-- </ul> -->

**Parent topic:**[Deploying and managing models](https://dataplatform.cloud.ibm.com/docs/content/wsj/wmls/wmls-deploy-overview.html)

<!-- </article "role="article" "> -->
# Deleting deployment spaces #

Delete existing deployment spaces that you don't require anymore\.

Important:Before you delete a deployment space, you must delete all the deployments that are associated with it\. Only a project admin can delete a deployment space\. For more information, see [Deployment space collaborator roles and permissions](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/collaborator-permissions-wml.html)\.

To remove a deployment space, follow these steps:

<!-- <ol> -->

1.  From the navigation menu, click **Deployments**\.
2.  In the deployments list, click the **Spaces** tab and find the deployment space that you want to delete\.
3.  Hover over the deployment space, select the menu (![Menu icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/images/open-close-icon.png)) icon, and click **Delete**\.
4.  In the confirmation dialog box, click **Delete**\.

<!-- </ol> -->

## Learn more ##

To learn more about how to clean up a deployment space and delete it programmatically, refer to:

<!-- <ul> -->

 *  [Notebook on managing machine learning artifacts](https://dataplatform.cloud.ibm.com/exchange/public/entry/view/e5e78be14e2260ccb4bcf8181d093d7b)
 *  [Notebook on managing spaces](https://dataplatform.cloud.ibm.com/exchange/public/entry/view/e5e78be14e2260ccb4bcf8181d0967e3)

<!-- </ul> -->

**Parent topic:**[Deployment spaces](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-spaces_local.html)

<!-- </article "role="article" "> -->
# Exporting space assets from deployment spaces #

You can export assets from a deployment space so that you can share the space with others or reuse the assets in another space\.

For a list of assets that you can export from space, refer to [Assets in a deployment space](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-space-add-assets-all.html)\.

## Exporting space assets from the UI ##

Important:To avoid problems with importing the space, export all dependencies together with the space\. For more information, see [Exporting a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/export-project.html)\.

To export space assets from the UI:

<!-- <ol> -->

1.  From your deployment space, click the import and export space (![Import or Export space icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/images/import-export-icon.png)) icon\. From the list, select **Export space**\.
2.  Click **New export file**\. Specify a file name and an optional description\.  
    **Tip:** To encrypt sensitive data in the exported archive, type the password in the **Password** field.
3.  Select the assets that you want to export with the space\.
4.  Click **Create** to create the export file\.
5.  After the space is exported, click the download (![Download icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/images/download-icon.png)) to save the file\.

<!-- </ol> -->

You can reuse this space by choosing **Create a space from a file** when you create a new space\.

## Learn more ##

<!-- <ul> -->

 *  [Importing spaces and projects into existing deployment spaces](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-space-import-to-space.html)\.

<!-- </ul> -->

**Parent topic:**[Deployment spaces](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-spaces_local.html)

<!-- </article "role="article" "> -->
# Deployment spaces #

Deployment spaces contain deployable assets, deployments, deployment jobs, associated input and output data, and the associated environments\. You can use spaces to deploy various assets and manage your deployments\.

Deployment spaces are not associated with projects\. You can publish assets from multiple projects to a space, and you can deploy assets to more than one space\. For example, you might have a test space for evaluating deployments, and a production space for deployments that you want to deploy in business applications\.

The deployments dashboard is an aggregate view of deployment activity available to you, across spaces\. For details, refer to [Deployments dashboard](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/operator-view.html)\.

When you open a space from the UI, you see these elements:

![Detailed information about a deployment space](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/images/DeploymentSpace.svg)

You can share a space with other people\. When you add collaborators to a deployment space, you can specify which actions they can do by assigning them access levels\. For details on space collaborator permissions, refer to [Deployment space collaborator roles and permissions](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/collaborator-permissions-wml.html)\.

## Learn more ##

<!-- <ul> -->

 *  [Creating deployment spaces](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-space-create.html)
 *  [Managing assets in a deployment space](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-space-add-assets-all.html)
 *  [Creating deployments from a space](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-deploy-general.html)
 *  [Exporting space assets](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-space-export.html)
 *  [Deleting deployment spaces](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-space-delete.html)

<!-- </ul> -->

<!-- </article "role="article" "> -->
# Jupyter Notebook editor #

The Jupyter Notebook editor is largely used for interactive, exploratory data analysis programming and data visualization\. Only one person can edit a notebook at a time\. All other users can access opened notebooks in view mode only, while they are locked\.

You can use the preinstalled open source libraries that come with the notebook runtime environments, add your own libraries, and benefit from the IBM libraries provided at no extra cost\.

When your notebooks are ready, you can create jobs to run the notebooks directly from the Jupyter Notebook editor\. Your job configurations can use environment variables that are passed to the notebooks with different values when the notebooks run\.

## Learn more ##

<!-- <ul> -->

 *  [Quick start: Analyze data in a Jupyter notebook](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/get-started-analyze.html)
 *  [Create notebooks in the Jupyter Notebook editor](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/creating-notebooks.html)
 *  [Runtime environments for notebooks](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/environments-parent.html)
 *  [Libraries and scripts](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/libraries.html)
 *  [Code and run notebooks](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/code-run-notebooks.html)
 *  [Schedule a notebook](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/create-job-nb-editor.html)
 *  [Share and publish notebooks](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/share-notebooks.html)

<!-- </ul> -->

**Parent topic:**[Notebooks and scripts](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/notebooks-and-scripts.html)

<!-- </article "role="article" "> -->
# Promoting an environment template to a space #

If you created an environment template and associated it with an asset that you promoted to a deployment space, you can also promote the environment template to the same space\. Promoting the environment template to the same space enables running the asset in the same environment that was used in the project\.

You can only promote environment templates that you created\.

To promote an environment template associated with an asset that you promoted to a deployment space:

<!-- <ol> -->

1.  From the **Manage** tab of your project on the **Environments** page under **Templates**, select the custom environment template and click **Actions > Promote**\.
2.  Select the space that you promoted your asset to as the target deployment space and optionally provide a description and tags\.

<!-- </ol> -->

**Parent topic:**[Environments](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/environments-parent.html)

<!-- </article "role="article" "> -->
# Using Python functions to work with Cloud Object Storage #

To access and work with data that is in IBM Cloud Object Storage, you can use Python functions from a notebook\.

With your IBM Cloud Object Storage credentials, you can access and load data from IBM Cloud Object Storage to use in a notebook\. This data can be any object of type file\-like\-object, for example, byte buffers or string buffers\. The data that you upload can reside in a different IBM Cloud Object Storage bucket than the project's bucket\.

You can also upload data from a local system into IBM Cloud Object Storage from within a notebook\. This data can be a compressed file or Pickle object\.

See [Working With IBM Cloud Object Storage In Python](https://medium.com/ibm-data-science-experience/working-with-ibm-cloud-object-storage-in-python-fe0ba8667d5f) for more information\.

## Learn more ##

<!-- <ul> -->

 *  Use [ibm\-watson\-studio\-lib for Python](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ws-lib-python.html) to interact with Watson Studio projects and project assets\. The library also contains functions that simplify fetching files from IBM Cloud Object Storage\.
 *  [Control access to COS buckets](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/cos_buckets.html)

<!-- </ul> -->

**Parent topic:**[Loading and accessing data in a notebook](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/load-and-access-data.html)

<!-- </article "role="article" "> -->
# Using Spark in RStudio #

Although the RStudio IDE cannot be started in a Spark with R environment runtime, you can use Spark in your R scripts and Shiny apps by accessing Spark kernels programmatically\. RStudio uses the `sparklyr` package to connect to Spark from R\. The `sparklyr` package includes a `dplyr` interface to Spark data frames as well as an R interface to Spark’s distributed machine learning pipelines\.

You can connect to Spark from RStudio:

<!-- <ul> -->

 *  By connecting to a Spark kernel that runs locally in the RStudio container in IBM Watson Studio

<!-- </ul> -->

RStudio includes sample code snippets that show you how to connect to a Spark kernel in your applications for both methods\.

To use Spark in RStudio after you have launched the IDE:

<!-- <ol> -->

1.  Locate the `ibm_sparkaas_demos` directory under your home directory and open it\. The directory contains the following R scripts:
    
    <!-- <ul> -->
    
     *  A readme with details on the included R sample scripts
     *  `spark_kernel_basic_local.R` includes sample code of how to connect to a local Spark kernel
     *  `spark_kernel_basic_remote.R` includes sample code of how to connect to a remote Spark kernel
     *  The files `sparkaas_flights.R`and `sparkaas_mtcars.R` are two examples of how to use Spark in a small sample application
    
    <!-- </ul> -->
    
2.  Use the sample code snippets in your R scripts or applications to help you get started using Spark\.

<!-- </ol> -->

## Connecting to Spark from RStudio ##

To connect to Spark from RStudio using the `Sparklyr` R package, you need a Spark with R environment\. You can either use the default Spark with R environment that is provided or create a custom Spark with R environment\. To create a custom environment, see [Creating environment templates](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/create-customize-env-definition.html)\.

Follow these steps after you launch RStudio in an RStudio environment:

Use the following sample code to get a listing of the Spark environment details and to connect to a Spark kernel from your RStudio session:

    # load spark R packages
    library(ibmwsrspark)
    library(sparklyr)
    
    # load kernels
    kernels <- load_spark_kernels()
    
    # display kernels
    display_spark_kernels()
    
    # get spark kernel Configuration
    
    conf <- get_spark_config(kernels[1])
    # Set spark configuration
    conf$spark.driver.maxResultSize <- "1G"
    # connect to Spark kernel
    
    sc <- spark_connect(config = conf)

Then to disconnect from Spark, use:

    # disconnect
    spark_disconnect(sc)

Examples of these commands are provided in the readme under `/home/wsuser/ibm_sparkaas_demos`\.

**Parent topic:**[RStudio](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/rstudio-overview.html)

<!-- </article "role="article" "> -->
# Compute resource options for AutoAI experiments in projects #

When you run an AutoAI experiment in a project, the type, size, and power of the hardware configuration available depend on the type of experiment you build\.

<!-- <ul> -->

 *  [Default hardware configurations](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/run-autoai.html?context=cdpaas&locale=en#default)
 *  [Compute usage](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/run-autoai.html?context=cdpaas&locale=en#compute)

<!-- </ul> -->

## Default hardware configurations ##

The type of hardware configuration available for your AutoAI experiment depends on the type of experiment you are building\. A standard AutoAI experiment, with a single data source, has a single, default hardware configuration\. An AutoAI experiment with joined data has options for increasing computational power\.

### Capacity units per hour for AutoAI experiments ###

<!-- <table> -->

Hardware configurations available in projects for AutoAI with a single data source

| Capacity type        | Capacity units per hour |
| -------------------- | ----------------------- |
| 8 vCPU and 32 GB RAM | 20                      |

<!-- </table ""> -->

The runtimes for AutoAI stop automatically when processing is complete\.

## Compute usage in projects ##

AutoAI consumes compute resources as CUH from the Watson Machine Learning service\.

You can monitor the total monthly amount of CUH consumption for the Watson Machine Learning service on the **Resource usage** page on the **Manage** tab of your project\.

## Learn more ##

<!-- <ul> -->

 *  [AutoAI](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/autoai-overview.html)
 *  [Watson Machine Learning service](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/wml-plans.html)
 *  [Compute resource options for assets and deployments in spaces](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/run-cuh-deploy-spaces.html)
 *  [Monitoring account resource usage](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/monitor-resources.html)

<!-- </ul> -->

**Parent topic:**[Choosing compute resources for tools](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/environments-parent.html)

<!-- </article "role="article" "> -->
# Compute resource options for Decision Optimization #

When you run a Decision Optimization model, you use the Watson Machine Learning instance that is linked to the deployment space associated with your experiment\.

<!-- <ul> -->

 *  [Default hardware configurations](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/run-decisionopt.html?context=cdpaas&locale=en#default)
 *  [Compute usage](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/run-decisionopt.html?context=cdpaas&locale=en#compute)

<!-- </ul> -->

## Default hardware configuration ##

The following hardware configuration is used by default when running models in an experiment:

<!-- <table> -->

| Capacity type       | Capacity units per hour (CUH) |
| ------------------- | ----------------------------- |
| 2 vCPU and 8 GB RAM | 6                             |

<!-- </table ""> -->

The CUH is consumed only when the model is running and not when you are adding data or editing your model\.

You can also switch to any other experiment environment as required\. See the [Decision Optimization plans](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/wml-plans.html#do) for a list of environments for Decision Optimization experiments\.

For more information on how to configure Decision Optimization experiment environments, see [Configuring environments](https://dataplatform.cloud.ibm.com/docs/content/DO/DODS_Introduction/configureEnvironments.html)\.

## Compute usage in projects ##

Decision Optimization experiments consume compute resources as CUH from the Watson Machine Learning service\.

You can monitor the total monthly amount of CUH consumption for the Watson Machine Learning service on the **Resource usage** page on the **Manage** tab of your project\.

## Learn more ##

<!-- <ul> -->

 *  [Decision Optimization](https://dataplatform.cloud.ibm.com/docs/content/DO/DOWS-Cloud_home.html)
 *  [Watson Machine Learning plans and compute usage](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/wml-plans.html)
 *  [Monitoring account resource usage](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/monitor-resources.html)

<!-- </ul> -->

**Parent topic:**[Choosing compute resources for tools](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/environments-parent.html)

<!-- </article "role="article" "> -->
# Compute resource options for Tuning Studio experiments in projects #

A Tuning Studio experiment has a single hardware configuration\.

The following table shows the hardware configuration that is used when tuning foundation models in a tuning experiment\.

<!-- <table> -->

Hardware configuration available in projects for Tuning Studio

| Capacity type | Capacity units per hour |
| ------------- | ----------------------- |

<!-- </table ""> -->

NVIDIA A100 80GB GPU\|43\|

## Compute usage in projects ##

Tuning Studio consumes compute resources as CUH from the Watson Machine Learning service\.

You can monitor the total monthly amount of CUH consumption for the Watson Machine Learning service on the **Resource usage** page on the **Manage** tab of your project\.

## Learn more ##

<!-- <ul> -->

 *  [Tuning Studio](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-tuning-studio.html)
 *  [Watson Machine Learning plans](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/wml-plans.html)
 *  [Compute resource options for assets and deployments in spaces](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/run-cuh-deploy-spaces.html)
 *  [Monitoring account resource usage](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/monitor-resources.html)

<!-- </ul> -->

**Parent topic:**[Choosing compute resources for tools](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/environments-parent.html)

<!-- </article "role="article" "> -->
# SPSS predictive analytics algorithms for scoring #

A PMML\-compliant scoring engine supports:

<!-- <ul> -->

 *  PMML\-compliant models (4\.2 and earlier versions) produced by various vendors, except for Baseline Model, ScoreCard Model, Sequence Model, and Text Model\. Refer to the [Data Mining Group (DMG) web site](http://www.dmg.org/) for a list of supported models\.
 *  Non\-PMML models produced by IBM SPSS products: Discriminant and Bayesian networks
 *  PMML 4\.2 transformations completely

<!-- </ul> -->

Different kinds of models can produce various scoring results\. For example:

<!-- <ul> -->

 *  Classification models (those with a categorical target: Bayes Net, General Regression, Mining, Naive Bayes, k\-Nearest Neighbor, Neural Network, Regression, Ruleset, Support Vector Machine, and Tree) produce:
    
    <!-- <ul> -->
    
     *  Predicted values
     *  Probabilities
     *  Confidence values
    
    <!-- </ul> -->
    
 *  Regression models (those with a continuous target: General Regression, Mining, k\-Nearest Neighbor, Neural Network, Regression, and Tree) produce predicted values; some also produce standard errors\.
 *  Cox regression (in General Regression) produces predicted survival probability and cumulative hazard values\.
 *  Tree models also produce Node ID\.
 *  Clustering models produce Cluster ID and Cluster affinity\.
 *  Anomaly Detection (represented as Clustering) produces anomaly index and top reasons\.
 *  Association models produce Consequent, Rule ID, and confidence for top matching rules\.

<!-- </ul> -->

**Python example code:**

    from spss.ml.score import Score
    
    with open("linear.pmml") as reader:
        pmmlString = reader.read()
    
    score = Score().fromPMML(pmmlString)
    scoredDf = score.transform(data)
    scoredDf.show()

**Parent topic:**[SPSS predictive analytics algorithms](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/spss-algorithms.html)

<!-- </article "role="article" "> -->
# SPSS predictive analytics algorithms #

You can use the following SPSS predictive analytics algorithms in your notebooks\. Code samples are provided for Python notebooks\.

Notebooks must run in a Spark with Python environment runtime\. To run the algorithms described in this section, you don't need the SPSS Modeler service\.

<!-- <ul> -->

 *  [Data preparation](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/datapreparation-guides.html)
 *  [Classification and regression](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/classificationandregression-guides.html)
 *  [Clustering](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/clustering-guides.html)
 *  [Forecasting](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/forecasting-guides.html)
 *  [Survival analysis](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/survivalanalysis-guides.html)
 *  [Score](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/score-guides.html)

<!-- </ul> -->

**Parent topic:**[Notebooks and scripts](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/notebooks-and-scripts.html)

<!-- </article "role="article" "> -->
# Compute resource options for Synthetic Data Generator in projects #

To create data with the Synthetic Data Generator, you must have the Watson Studio and Watson Machine Learning services provisioned\. Running a synthetic data flow consumes compute resources from the Watson Studio service\.

### Capacity units per hour for Synthetic Data Generator ###

<!-- <table> -->

| Capacity type       | Capacity units per hour |
| ------------------- | ----------------------- |
| 2 vCPU and 8 GB RAM | 7                       |

<!-- </table ""> -->

## Compute usage in projects ##

Running a synthetic data flow consumes compute resources from the Watson Studio service\.

You can monitor the total monthly amount of CUH consumption for Watson Studio on the **Resource usage** page on the **Manage** tab of your project\.

## Learn more ##

<!-- <ul> -->

 *  [Synthetic data](https://dataplatform.cloud.ibm.com/docs/content/wsj/synthetic/synthetic_data_overview_sd.html)
 *  [Watson Machine Learning service](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/wml-plans.html)
 *  [Watson Studio service](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/ws-plans.html)
 *  [Monitoring account resource usage](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/monitor-resources.html)

<!-- </ul> -->

**Parent topic:**[Choosing compute resources for tools](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/environments-parent.html)

<!-- </article "role="article" "> -->