itsjhuang/watsonx-docs-document-type huggingface dataset:
applytwostepnode properties
applytwostepnode properties You can use TwoStep modeling nodes to generate a TwoStep model nugget. The scripting name of this model nugget is applytwostepnode. For more information on scripting the modeling node itself, see [twostepnode properties](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/scripting_guide/clementine/twostepnodeslots.htmltwostepnodeslots). applytwostepnode properties Table 1. applytwostepnode properties applytwostepnode Properties Values Property description enable_sql_generation udf <br>native When using data from a database, SQL code can be pushed back to the database for execution, providing superior performance for many operations.
Adding data to Data Refinery
Adding data to Data Refinery After you [create a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/projects.html) and you [create connections](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/create-conn.html) or you [add data assets](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/add-data-project.html) to the project, you can then add data to Data Refinery and start prepping that data for analysis. You can add data to Data Refinery in one of several ways: * Select Prepare data from the overflow menu (![overflow menu](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/images/actions.png)) of a data asset in the All assets list for the project * Preview a data asset in the project and then click Prepare data * Navigate to Data Refinery first and then add data to it Navigate to Data Refinery 1. Access Data Refinery from within a project. Click the Assets tab. 2. Click New asset > Prepare and visualize data. 3. Select the data that you want to work with from Data assets or from Connections. From Data assets: * Select a data file (the selection includes data files that were already shaped with Data Refinery) * Select a connected data asset From Connections: * Select a connection and file * Select a connection, folder, and file * Select a connection, schema, and table or view Data Refinery supports these file types: Avro, CSV, delimited text files, JSON, Microsoft Excel (xls and xlsx formats. First sheet only, except for connections and connected data assets.), Parquet, SAS with the "sas7bdat" extension (read only), TSV (read only) Data Refinery operates on a sample subset of rows in the data set. The sample size is 1 MB or 10,000 rows, whichever comes first. However, when you run a job for the Data Refinery flow, the entire data set is processed. If the Data Refinery flow fails with a large data asset, see workarounds in [Troubleshooting Data Refinery](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/ts_index.html). Data connections marked with a key icon (![the key symbol for private connections](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/images/privatekey.png)) are locked. If you are authorized to access the data source, you are asked to enter your personal credentials the first time you select it. This one-time step permanently unlocks the connection for you. After you have unlocked the connection, the key icon is no longer displayed. See [Adding connections to projects](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/create-conn.html). 4. Click Add to load the data into Data Refinery. Next steps * [Refine your data](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/refining_data.html) * [Validate your data](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/metrics.html) * [Use visualizations to gain insights into your data](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/visualizations.html) Parent topic:[Refining data](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/refining_data.html)
Compute resource options for Synthetic Data Generator in projects
Compute resource options for Synthetic Data Generator in projects To create data with the Synthetic Data Generator, you must have the Watson Studio and Watson Machine Learning services provisioned. Running a synthetic data flow consumes compute resources from the Watson Studio service. Capacity units per hour for Synthetic Data Generator Capacity type Capacity units per hour 2 vCPU and 8 GB RAM 7 Compute usage in projects Running a synthetic data flow consumes compute resources from the Watson Studio service. You can monitor the total monthly amount of CUH consumption for Watson Studio on the Resource usage page on the Manage tab of your project. Learn more * [Synthetic data](https://dataplatform.cloud.ibm.com/docs/content/wsj/synthetic/synthetic_data_overview_sd.html) * [Watson Machine Learning service](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/wml-plans.html) * [Watson Studio service](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/ws-plans.html) * [Monitoring account resource usage](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/monitor-resources.html) Parent topic:[Choosing compute resources for tools](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/environments-parent.html)
Managing all projects in the account
Managing all projects in the account If you have the required permission, you can view and manage all projects in your IBM Cloud account. You can add yourself to a project so that you can delete it or change its collaborators. Requirements To manage all projects in the account, you must: * Restrict resources to the current account. See steps to [set the scope for resources](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/account-settings.htmlset-the-scope-for-resources). * Have the Manage projects permission that is provided by the IAM Manager role for the IBM Cloud Pak for Data service. Assigning the Manage projects permission To grant the Manage projects permission to a user who is already in your IBM Cloud account: 1. From the navigation menu, choose Administration > Access (IAM) to open the Manage access and users page in your IBM Cloud account. 2. Select the user on the Users page. 3. Click the Access tab and then choose Assign access+. 4. Select Access policy. 5. For Service, choose IBM Cloud Pak for Data. 6. For Service access, select the Manager role. 7. For Platform access, assign the Editor role. 8. Click Add and Assign to assign the policy to the user. Managing projects You can add yourself to a project when you need to delete the project, delete collaborators, or assign the Admin role to a collaborator in the project. To manage projects: * View all active projects on the Projects page in IBM watsonx by clicking the drop-down menu next to the search field and selecting All active projects. * Join any project as Admin by clicking Join as admin in the Your role column. * Filter projects to identify which projects you are not a collaborator in, by clicking the filter icon ![Filter icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/images/filter.svg) and selecting Your role > No membership. For more details on managing projects, see [Administering projects](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/admin-project.html). Learn more * [Administering projects](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/admin-project.html) Parent topic:[Managing IBM watsonx](https://dataplatform.cloud.ibm.com/docs/content/wsj/console/wdp_admin_console.html)
Defining the targets (SPSS Modeler)
Defining the targets 1. Add a Type node after the Filler node, then double-click the Type node to open its properties. 2. Set the role to None for the DATE_ field. Set the role to Target for all other fields (the Market_n fields plus the Total field). 3. Click Read Values to populate the Values column. Figure 1. Setting the role for fields ![Setting the role for fields](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/tut_bandwidth_forecast_targets.png)
Handling Pipeline errors
Handling Pipeline errors You can specify how to respond to errors in a pipeline globally, with an error policy, and locally, by overriding the policy on the node level. You can also create a custom error-handling response. Setting global error policy The error policy sets the default behavior for errors in a pipeline. You can override this behavior for any node in the pipeline. To set the global error policy: 1. Click the Manage default settings icon on the toolbar. 2. Choose the default response to an error under the Error policy: * Fail pipeline on error stops the flow and initiates an error-handling flow. * Continue pipeline on error tries to continue running the pipeline. Note: Continue pipeline on error affects nodes that use the default error policy and does not affect node-specific error policies. 3. You can optionally create a custom error-handling response for a flow failure. Specifying an error response If you opt for Fail pipeline on error for either the global error policy or for a node-specific policy, you can further specify what happens on failure. For example, if you check the Show icon on nodes that are linked to an error-handling pipeline, an icon flags a node with an error to help debug the flow. Specifying a node-specific error policy You can override the default error policy for any node in the pipeline. 1. Click a node to open the configuration pane. 2. Check the option to Override default error policy with: * Fail pipeline on error * Continue pipeline on error Viewing all node policies To view all node-specific error handling for a pipeline: 1. Click Manage default settings on the toolbar. 2. Click the view all node policies link under Error policy. A list of all nodes in the pipeline show which nodes use the default policy, and which override the default policy. Click a node name to see the policy details. Use the view filter to show: * All error policies: all nodes * Default policy: all nodes that use the default policy * Override default policy: all nodes that override the default policy * Fail pipeline on error: all nodes that stop the flow on error * Continue pipeline on error: all nodes that try to continue the flow on error Running the Fail on error flow If you specify that the flow fails on error, a secondary error handling flow starts when an error is encountered. Adding a custom error response If Create custom error handling response is checked on default settings for error policy, you can add an error handling node to the canvas so you can configure a custom error response. The response applies to all nodes configured to fail when an error occurs. Parent topic:[Creating a pipeline](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-orchestration-create.html)
applyneuralnetworknode properties
applyneuralnetworknode properties You can use Neural Network modeling nodes to generate a Neural Network model nugget. The scripting name of this model nugget is applyneuralnetworknode. For more information on scripting the modeling node itself, see [neuralnetworknode properties](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/scripting_guide/clementine/properties/neuralnetworkslots.htmlneuralnetworkslots). applyneuralnetworknode properties Table 1. applyneuralnetworknode properties applyneuralnetworknode Properties Values Property description use_custom_name flag custom_name string confidence onProbability <br>onIncrease score_category_probabilities flag max_categories number score_propensity flag enable_sql_generation false <br>true <br>native When using data from a database, SQL code can be pushed back to the database for execution, providing superior performance for many operations.
Building a model (SPSS Modeler)
Building a model By exploring and manipulating the data, you have been able to form some hypotheses. The ratio of sodium to potassium in the blood seems to affect the choice of drug, as does blood pressure. But you cannot fully explain all of the relationships yet. This is where modeling will likely provide some answers. In this case, you will try to fit the data using a rule-building model called C5.0. Since you're using a derived field, Na_to_K, you can filter out the original fields, Na and K, so they're not used twice in the modeling algorithm. You can do this by using a Filter node. 1. Place a Filter node on the canvas and connect it to the Derive node. Figure 1. Filter node ![Derive node](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/tut_drug_build_flow.png) 2. Double-click the Filter node to edit its properties. Name it Discard Fields. 3. For Mode, make sure Filter the selected fields is selected. Then select the K and Na fields. Click Save. 4. Place a Type node on the canvas and connect it to the Filter node. With the Type node, you can indicate the types of fields you're using and how they're used to predict the outcomes. Figure 2. Type node ![Type node](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/tut_drug_build_flow2.png) 5. Double-click the Type node to edit its properties. Name it Define Types. 6. Set the role for the Drug field to Target, indicating that Drug is the field you want to predict. Leave the role for the other fields set to Input so they'll be used as predictors. Click Save. 7. To estimate the model, place a C5.0 node on the canvas and attach it to the end of the flow. Then click the Run button on the toolbar to run the flow.
{{ document.title.text }}
Extraction attack ![icon for robustness risk](https://dataplatform.cloud.ibm.com/docs/content/wsj/ai-risk-atlas/images/risk-robustness.svg)Risks associated with inputInferenceRobustnessAmplified Description An attack that attempts to copy or steal the AI model by appropriately sampling the input space, observing outputs, and building a surrogate model, is known as an extraction attack. Why is extraction attack a concern for foundation models? A successful attack mimics the model, enabling the attacker to repurpose it for their benefit such as eliminating a competitive advantage or causing reputational harm. Parent topic:[AI risk atlas](https://dataplatform.cloud.ibm.com/docs/content/wsj/ai-risk-atlas/ai-risk-atlas.html)
Evaluation charts
Evaluation charts Evaluation charts are similar to histograms or collection graphs. Evaluation charts show how accurate models are in predicting particular outcomes. They work by sorting records based on the predicted value and confidence of the prediction, splitting the records into groups of equal size (quantiles), and then plotting the value of the criterion for each quantile, from highest to lowest. Multiple models are shown as separate lines in the plot. Outcomes are handled by defining a specific value or range of values as a "hit". Hits usually indicate success of some sort (such as a sale to a customer) or an event of interest (such as a specific medical diagnosis). Flag : Output fields are straightforward; hits correspond to true values. Nominal : For nominal output fields, the first value in the set defines a hit. Continuous : For continuous output fields, hits equal values greater than the midpoint of the field's range. Evaluation charts can also be cumulative so that each point equals the value for the corresponding quantile plus all higher quantiles. Cumulative charts usually convey the overall performance of models better, whereas noncumulative charts often excel at indicating particular problem areas for models.
Testing (SPSS Modeler)
Testing Both of the generated model nuggets are connected to the Type node. 1. Reposition the nuggets as shown, so the Type node connects to the neural net nugget, which connects to the C5.0 nugget. 2. Attach an Analysis node to the C5.0 nugget. 3. Edit the Data Asset node to use the file cond2n.csv (instead of cond1n.csv), which contains unseen test data. 4. Right-click the Analysis node and select Run. Doing so yields figures reflecting the accuracy of the trained network and rule. Figure 1. Testing the trained network ![Testing the trained network](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/tut_condition_analysis.png)
Deploying and managing assets
Deploying and managing assets Use Watson Machine Learning to deploy models and solutions so that you can put them into productive use, then monitor the deployed assets for fairness and explainability. You can also automate the AI lifecycle to keep your deployed assets current. Completing the AI lifecycle After you prepare your data and build then train models or solutions, you complete the AI lifecycle by deploying and monitoring your assets. ![Overview of model workflow](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/images/ml-engineer-overview-wx.svg) Deployment is the final stage of the lifecycle of a model or script, where you run your models and code. Watson Machine Learning provides the tools that you need to deploy an asset, such as a predictive model, Python function. You can also deploy foundation model assets, such as prompt templates, to put them into production. Following deployment, you can use model management tools to evaluate your models. IBM Watson OpenScale tracks and measures outcomes from your AI models, and helps ensure they remain fair, explainable, and compliant. Watson OpenScale also detects and helps correct the drift in accuracy when an AI model is in production. Finally, you can use IBM Watson Pipelines to manage your ModelOps processes. Create a pipeline that automates parts of the AI lifecycle, such as training and deploying a machine learning model. Next steps * To learn more about how to manage assets in a deployment space, see [Manage assets in a deployment space](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-spaces_local.html). * To learn more about how to deploy assets from a deployment space, see [Deploy assets from a deployment space](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-deploy-general.html). * To learn more how to deploy by using [Python client](https://ibm.github.io/watson-machine-learning-sdk/) or , see [Sample notebook](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-samples-overview.html).
Defining missing values (SPSS Modeler)
Defining missing values In the Type node settings, select the desired field in the table and then click the gear icon at the end of its row. Missing values settings are available in the window that appears. Select Define missing values to define missing value handing for this field. Here you can define explicit values to be considered as missing values for this field, or this can also be accomplished by means of a downstream Filler node.
{{ document.title.text }}
Trust calibration ![icon for value alignment risk](https://dataplatform.cloud.ibm.com/docs/content/wsj/ai-risk-atlas/images/risk-value-alignment.svg)Risks associated with outputValue alignmentNew Description Trust calibration presents problems when a person places too little or too much trust in an AI model's guidance, resulting in poor decision making. Why is trust calibration a concern for foundation models? In tasks where humans make choices based on AI-based suggestions, consequences of poor decision making increase with the importance of the decision. Bad decisions can harm users and can lead to financial harm, reputational harm, and other legal consequences for business entities. Parent topic:[AI risk atlas](https://dataplatform.cloud.ibm.com/docs/content/wsj/ai-risk-atlas/ai-risk-atlas.html)
Building CLEM expressions (SPSS Modeler)
Building CLEM (legacy) expressions
Creating synthetic data from production data
Creating synthetic data from production data Using the Synthetic Data Generator graphical editor flow tool, you can generate a structured synthetic data set based on your production data. You can import data, anonymize, mimic (to generate synthetic data), export, and review your data. Before you can use mimic and mask to create synthetic data, you need [to create a task](https://dataplatform.cloud.ibm.com/docs/content/wsj/synthetic/synthetic_data_overview_sd.htmlcreate-synthetic). 1. The Generate synthetic tabular data flow window opens. Select use case Leverage your existing data. Click Next. ![Generate synthetic tabular data flow window](https://dataplatform.cloud.ibm.com/docs/content/wsj/synthetic/images/sd-mimic-mask-flow.png) 2. Select Import data. You can also drag-and-drop a data file into your project. You can also select data from a project. For more information, see [Importing data](https://dataplatform.cloud.ibm.com/docs/content/wsj/synthetic/import_data_sd.html). ![Import data](https://dataplatform.cloud.ibm.com/docs/content/wsj/synthetic/images/sd-import-data.png) 3. Once you have imported your data, you can use the Synthetic Data Generator graphical flow editor tool to anonymize your production data, masking the data. You can disguise column names, column values, or both, when working with data that is to be included in a model downstream of the node. For example, you can use bank customer data and hide marital status. ![Anonymize data](https://dataplatform.cloud.ibm.com/docs/content/wsj/synthetic/images/sd-anonymize.png) 4. You can then use the Synthetic Data Generator tool to mimic your production data. This will generate synthetic data, based on your production data, using a set of candidate statistical distributions to modify each column in your data. ![Mimic data](https://dataplatform.cloud.ibm.com/docs/content/wsj/synthetic/images/sd-mimic-options.png) 5. You can export your synthetic data and review it. For more information, see [Exporting synthetic data](https://dataplatform.cloud.ibm.com/docs/content/wsj/synthetic/export_data_sd.html). ![Export data](https://dataplatform.cloud.ibm.com/docs/content/wsj/synthetic/images/sd-export.png) Learn more [Creating synthetic data from a custom data schema](https://dataplatform.cloud.ibm.com/docs/content/wsj/synthetic/generate_data_sd.html)
{{ document.title.text }}
Data poisoning ![icon for robustness risk](https://dataplatform.cloud.ibm.com/docs/content/wsj/ai-risk-atlas/images/risk-robustness.svg)Risks associated with inputTraining and tuning phaseRobustnessTraditional Description Data poisoning is a type of adversarial attack where an adversary or malicious insider injects intentionally corrupted, false, misleading, or incorrect samples into the training or fine-tuning dataset. Why is data poisoning a concern for foundation models? Poisoning data can make the model sensitive to a malicious data pattern and produce the adversary’s desired output. It can create a security risk where adversaries can force model behavior for their own benefit. In addition to producing unintended and potentially malicious results, a model misalignment from data poisoning can result in business entities facing legal consequences or reputational harms. Parent topic:[AI risk atlas](https://dataplatform.cloud.ibm.com/docs/content/wsj/ai-risk-atlas/ai-risk-atlas.html)