itsjhuang/watsonx-docs-document-type huggingface dataset:
Reducing input data string length (SPSS Modeler)
Reducing input data string length For binomial logistic regression, and auto classifier models that include a binomial logistic regression model, string fields are limited to a maximum of eight characters. Where strings are more than eight characters, you can recode them using a Reclassify node. This example uses the flow named Reducing Input Data String Length, available in the example project . The data file is drug_long_name.csv. This example focuses on a small part of a flow to show the type of errors that may be generated with overlong strings, and explains how to use the Reclassify node to change the string details to an acceptable length. Although the example uses a binomial Logistic Regression node, it is equally applicable when using the Auto Classifier node to generate a binomial Logistic Regression model.
{{ document.title.text }}
Inaccessible training data ![icon for explainability risk](https://dataplatform.cloud.ibm.com/docs/content/wsj/ai-risk-atlas/images/risk-explainability.svg)Risks associated with outputExplainabilityAmplified Description Without access to the training data, the types of explanations a model can provide are limited and more likely to be incorrect. Why is inaccessible training data a concern for foundation models? Low quality explanations without source data make it difficult for users, model validators, and auditors to understand and trust the model. Parent topic:[AI risk atlas](https://dataplatform.cloud.ibm.com/docs/content/wsj/ai-risk-atlas/ai-risk-atlas.html)
Specifying values for collection data (SPSS Modeler)
Specifying values for collection data Collection fields display non-geospatial data that's in a list. The only item you can set for the Collection measurement level is the List measure. By default, this measure is set to Typeless, but you can select another value to set the measurement level of the elements within the list. You can choose one of the following options: * Typeless * Continuous * Nominal * Ordinal * Flag
Managing data for model evaluations in watsonx.governance
Managing data for model evaluations in watsonx.governance To enable model evaluations in watsonx.governance, you must prepare your data for logging to generate insights. You must provide your model data to watsonx.governance in a format that it supports to enable model evaluations. watsonx.governance processes your model transactions and logs the data in the watsonx.governance data mart. The data mart is the logging database that stores the data that is used for model evaluations. The following sections describe the different types of data that watsonx.governance logs for model evaluations: Payload data Payload data contains the input and output transactions for your deployment. To configure explainability and fairness and drift evaluations, watsonx.governance must receive payload data from your model that it stores in a payload logging table. The payload logging table contains the feature and prediction columns that exist in your training data and a prediction probability column that contains the model's confidence in the prediction that it provides. The table also includes timestamp and ID columns to identify each scoring request that you send to watsonx.governance as shown in the following example: ![Python SDK sample output of payload logging table](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/images/wos-ntbok.png) You must send scoring requests to provide watsonx.governance with a log of your model transactions. For more information, see [Managing payload data](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-payload-logging.html). Feedback data Feedback data is labeled data that matches the structure of training data and includes known model outcomes that are compared to your model predictions to measure the accuracy of your model. watsonx.governance uses feedback data to enable you to configure quality evaluations. You must upload feedback data regularly to watsonx.governance to continuously measure the accuracy of your model predictions. For more information, see [Managing feedback data](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-manage-feedback-data.html). Learn more [Sending model transactions](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-send-model-transactions.html)
Defining class attributes and methods
Defining class attributes and methods Any variable that's bound in a class is a class attribute. Any function defined within a class is a method. Methods receive an instance of the class, conventionally called self, as the first argument. For example, to define some class attributes and methods, you might enter the following script: class MyClass attr1 = 10 class attributes attr2 = "hello" def method1(self): print MyClass.attr1 reference the class attribute def method2(self): print MyClass.attr2 reference the class attribute def method3(self, text): self.text = text instance attribute print text, self.text print my argument and my attribute method4 = method3 make an alias for method3 Inside a class, you should qualify all references to class attributes with the class name (for example, MyClass.attr1). All references to instance attributes should be qualified with the self variable (for example, self.text). Outside the class, you should qualify all references to class attributes with the class name (for example, MyClass.attr1) or with an instance of the class (for example, x.attr1, where x is an instance of the class). Outside the class, all references to instance variables should be qualified with an instance of the class (for example, x.text).
Decision Optimization notebook tutorial
Solving and analyzing a model: the diet problem This example shows you how to create and solve a Python-based model by using a sample. Procedure To create and solve a Python-based model by using a sample: 1. Download and extract all the [DO-samples](https://github.com/IBMDecisionOptimization/DO-Samples) on to your computer. You can also download just the diet.zip file from the Model_Builder subfolder for your product and version, but in this case, do not extract it. 2. Open your project or create an empty project. 3. On the Manage tab of your project, select the Services and integrations section and click Associate service. Then select an existing Machine Learning service instance (or create a new one ) and click Associate. When the service is associated, a success message is displayed, and you can then close the Associate service window. 4. Select the Assets tab. 5. Select New asset > Solve optimization problems in the Work with models section. 6. Click Local file in the Solve optimization problems window that opens. 7. Browse to find the Model_Builder folder in your downloaded DO-samples. Select the relevant product and version subfolder. Choose the Diet.zip file and click Open. Alternatively use drag and drop. 8. If you haven't already associated a Machine Learning service with your project, you must first select Add a Machine Learning service to select or create one before you choose a deployment space for your experiment. 9. Click New deployment space, enter a name, and click Create (or select an existing space from the drop-down menu). 10. Click Create.A Decision Optimization model is created with the same name as the sample. 11. In the Prepare data view, you can see the data assets imported.These tables represent the min and max values for nutrients in the diet (diet_nutrients), the nutrients in different foods (diet_food_nutrients), and the price and quantity of specific foods (diet_food). ![Tables of input data in Prepare data view](https://dataplatform.cloud.ibm.com/docs/content/DO/DODS_Introduction/images/Cloudpreparedata2.png) 12. Click Build model in the sidebar to view your model.The Python model minimizes the cost of the food in the diet while satisfying minimum nutrient and calorie requirements. ![Python model for diet problem displayed in Run model view](https://dataplatform.cloud.ibm.com/docs/content/DO/DODS_Introduction/images/newrunmodel3.png) Note also how the inputs (tables in the Prepare data view) and the outputs (in this case the solution table to be displayed in the Explore solution view) are specified in this model. 13. Run the model by clicking the Run button in the Build model view.
Decision Optimization output data definition
Output data definition When submitting your job you can define what output data you want and how you collect it (as either inline or referenced data). For more information about output file types and names see [Model input and output data file formats](https://dataplatform.cloud.ibm.com/docs/content/DO/WML_Deployment/ModelIOFileFormats.htmltopic_modelIOFileFormats). Some output data definition examples: * To collect solution.csv output as inline data: "output_data": [{ "id":"solution.csv" }] * Regexp can be also used as an identifier. For example to collect all csv output files as inline data: "output_data": [{ "id":"..csv" }] * Similarly for reference data, to collect all csv files in COS/S3 in job specific folder, you can combine regexp and ${job_id} and ${ attachment_name } place holder "output_data_references": [{ "id":"..csv", "type": "connection_asset", "connection": { "id" : <connection_guid> }, "location": { "bucket": "XXXXXXXXX", "path": "${job_id}/${attachment_name}" } }] For example, here if you have a job with identifier <XXXXXXXXX> to generate a solution.csv file, you will have in your COS/S3 bucket, a XXXXXXXXX / solution.csv file.
Monitoring the experiment and saving the model
Monitoring the experiment and saving the model Any party or admin with collaborator access to the experiment can monitor the experiment and save a copy of the model. As the experiment runs, you can check the progress of the experiment. After the training is complete, you can view your results, save and deploy the model, and then test the model with new data. Monitoring the experiment When all parties run the party connector script, the experiment starts training automatically. As the training runs, you can view a dynamic diagram of the training progress. For each round of training, you can view the four stages of a training round: * Sending model: Federated Learning sends the model metrics to each party. * Training: The process of training the data locally. Each party trains to produce a local model that is fused. No data is exchanged between parties. * Receiving models: After training is complete, each party sends its local model to the aggregator. The data is not sent and remains private. * Aggregating: The aggregator combines the models that are sent by each of the remote parties to create an aggregated model. Saving your model When the training is complete, a chart that displays the model accuracy over each round of training is drawn. Hover over the points on the chart for more information on a single point's exact metrics. A Training rounds table shows details for each training round. The table displays the participating parties' average accuracy of their model training for each round. ![Screenshot of View Setup Information](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/images/fl-display.png) When you are done with the viewing, click Save model to project to save the Federated Learning model to your project. Rerun the experiment You can rerun the experiment as many times as you need in your project. Note:If you encounter errors when rerunning an experiment, see [Troubleshoot](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fl-troubleshoot.html) for more details. Deploying your model After you save your Federated Learning model, you can deploy and score the model like other machine learning models in a Watson Studio platform. See [Deploying models](https://dataplatform.cloud.ibm.com/docs/content/wsj/wmls/wmls-deploy-overview.html) for more details. Parent topic:[Creating a Federated Learning experiment](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fl-start.html)
Collection node (SPSS Modeler)
Collection node Collections are similar to histograms, but collections show the distribution of values for one numeric field relative to the values of another, rather than the occurrence of values for a single field. A collection is useful for illustrating a variable or field whose values change over time. Using 3-D graphing, you can also include a symbolic axis displaying distributions by category. Two-dimensional collections are shown as stacked bar charts, with overlays where used.
Restructure node (SPSS Modeler)
Restructure node With the Restructure node, you can generate multiple fields based on the values of a nominal or flag field. The newly generated fields can contain values from another field or numeric flags (0 and 1). The functionality of this node is similar to that of the Set to Flag node. However, it offers more flexibility by allowing you to create fields of any type (including numeric flags), using the values from another field. You can then perform aggregation or other manipulations with other nodes downstream. (The Set to Flag node lets you aggregate fields in one step, which may be convenient if you are creating flag fields.) Figure 1. Restructure node ![Restructure node](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/restructure_node.png)
Series array charts
Series array charts Series array charts include individual sub charts and display the Y-axis for all sub charts in the legend.
Importing a project
Importing a project You can create a project that is preloaded with assets by importing the project. Requirements A local file of a previously exported project : Importing a project from a local file is a method of copying a project. You can import a project from a file on your local system only if the ZIP file that you select was exported from a IBM watsonx project as a compressed file. You can import only projects that you exported from watsonx.ai. You cannot import a compressed file that was exported from a Cloud Pak for Data as a Service project. : If the exported file that you select to import was encrypted, you must enter the password that was used for encryption to enable decrypting sensitive connection properties. A sample project from Samples : You can create a project [from a project sample](https://dataplatform.cloud.ibm.com/samples?context=wx) to learn how to work with data in tools, such as notebooks to prepare data, analyze data, build and train models, and visualize analysis results. : The sample projects show how to accomplish goals, for example, to load and explore data, to create and train machine learning models for predictive analysis. Each project includes the required assets, such as notebooks, and all the data sets that you need to complete the example use case. Importing a project from a local file or sample To import a project: 1. Click New project on the home page or on your Projects page. 2. Choose whether to create a project based on an exported project file or a sample project. 3. Upload a project file or select a sample project. 4. On the New project screen, add a name and optional description for the project. 5. If the project file that you select to import is encrypted, you must enter the password that was used for encryption to enable decrypting sensitive connection properties. If you enter an incorrect password, the project file imports successfully, but sensitive connection properties are falsely decrypted. 6. Select the Restrict who can be a collaborator checkbox to restrict collaborators to members of your organization. You can't change this setting after you create the project. 7. Choose an existing [object storage service instance](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/storage-options.html) or create a new one. 8. Click Create. You can start adding resources if your project is empty, or begin working with the resources you imported. Learn more * [Administering a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/admin-project.html) * [Exporting project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/export-project.html) Parent topic:[Creating a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/projects.html)
Scripting and automation
Scripting and automation Scripting in SPSS Modeler is a powerful tool for automating processes in the user interface. Scripts can perform the same types of actions that you perform with a mouse or a keyboard, and you can use them to automate tasks that would be highly repetitive or time consuming to perform manually. You can use scripts to: * Impose a specific order for node executions in a flow. * Set properties for a node as well as perform derivations using a subset of CLEM (Control Language for Expression Manipulation). * Specify an automatic sequence of actions that normally involves user interaction—for example, you can build a model and then test it. * Set up complex processes that require substantial user interaction—for example, cross-validation procedures that require repeated model generation and testing. * Set up processes that manipulate flows—for example, you can take a model training flow, run it, and produce the corresponding model-testing flow automatically.
Watson Machine Learning on IBM watsonx
Watson Machine Learning on IBM watsonx Watson Machine Learning is part of IBM® watsonx.ai. Watson Machine Learning provides a full range of tools for your team to build, train, and deploy Machine Learning models. You can choose the tool with the level of automation or autonomy that matches your needs. Watson Machine Learning provides the following tools: * AutoAI experiment builder for automatically processing structured data to generate model-candidate pipelines. The best-performing pipelines can be saved as a machine learning model and deployed for scoring. * Deployment spaces give you the tools to view and manage model deployments. * Tools to view and manage model deployments.
Neural Net node (SPSS Modeler)
Neural Net node A neural network can approximate a wide range of predictive models with minimal demands on model structure and assumption. The form of the relationships is determined during the learning process. If a linear relationship between the target and predictors is appropriate, the results of the neural network should closely approximate those of a traditional linear model. If a nonlinear relationship is more appropriate, the neural network will automatically approximate the "correct" model structure. The trade-off for this flexibility is that the neural network is not easily interpretable. If you are trying to explain an underlying process that produces the relationships between the target and predictors, it would be better to use a more traditional statistical model. However, if model interpretability is not important, you can obtain good predictions using a neural network. Field requirements. There must be at least one Target and one Input. Fields set to Both or None are ignored. There are no measurement level restrictions on targets or predictors (inputs). The initial weights assigned to neural networks during model building, and therefore the final models produced, depend on the order of the fields in the data. Watsonx.ai automatically sorts data by field name before presenting it to the neural network for training. This means that explicitly changing the order of the fields in the data upstream will not affect the generated neural net models when a random seed is set in the model builder. However, changing the input field names in a way that changes their sort order will produce different neural network models, even with a random seed set in the model builder. The model quality will not be affected significantly given different sort order of field names.
Hidden variables
Hidden variables You can hide data by creating Private variables. Private variables can be accessed only by the class itself. If you declare names of the form __xxx or __xxx_yyy, that is with two preceding underscores, the Python parser will automatically add the class name to the declared name, creating hidden variables. For example: class MyClass: __attr = 10 private class attribute def method1(self): pass def method2(self, p1, p2): pass def __privateMethod(self, text): self.__text = text private attribute Unlike in Java, in Python all references to instance variables must be qualified with self; there's no implied use of this.
{{ document.title.text }}
Prompt injection ![icon for robustness risk](https://dataplatform.cloud.ibm.com/docs/content/wsj/ai-risk-atlas/images/risk-robustness.svg)Risks associated with inputInferenceRobustnessNew Description A prompt injection attack forces a model to produce unexpected output due to the structure or information contained in prompts. Why is prompt injection a concern for foundation models? Injection attacks can be used to alter model behavior and benefit the attacker. If not properly controlled, business entities could face fines, reputational harm, and other legal consequences. Parent topic:[AI risk atlas](https://dataplatform.cloud.ibm.com/docs/content/wsj/ai-risk-atlas/ai-risk-atlas.html)
Refression (SPSS Modeler)
Regression node Linear regression is a common statistical technique for classifying records based on the values of numeric input fields. Linear regression fits a straight line or surface that minimizes the discrepancies between predicted and actual output values. Requirements. Only numeric fields can be used in a regression model. You must have exactly one target field (with the role set to Target) and one or more predictors (with the role set to Input). Fields with a role of Both or None are ignored, as are non-numeric fields. (If necessary, non-numeric fields can be recoded using a Derive node. ) Strengths. Regression models are relatively simple and give an easily interpreted mathematical formula for generating predictions. Because regression modeling is a long-established statistical procedure, the properties of these models are well understood. Regression models are also typically very fast to train. The Regression node provides methods for automatic field selection in order to eliminate nonsignificant input fields from the equation. Note: In cases where the target field is categorical rather than a continuous range, such as yes/no or churn/don't churn, logistic regression can be used as an alternative. Logistic regression also provides support for non-numeric inputs, removing the need to recode these fields. See [Logistic node](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/logreg.htmllogreg) for more information.
Time (SPSS Modeler)
Time The CLEM language supports the time formats listed in this section. CLEM language time formats Table 1. CLEM language time formats Format Examples HHMMSS 120112, 010101, 221212 HHMM 1223, 0745, 2207 MMSS 5558, 0100 HH:MM:SS 12:01:12, 01:01:01, 22:12:12 HH:MM 12:23, 07:45, 22:07 MM:SS 55:58, 01:00 (H)H:(M)M:(S)S 12:1:12, 1:1:1, 22:12:12 (H)H:(M)M 12:23, 7:45, 22:7 (M)M:(S)S 55:58, 1:0 HH.MM.SS 12.01.12, 01.01.01, 22.12.12 HH.MM 12.23, 07.45, 22.07 MM.SS 55.58, 01.00 (H)H.(M)M.(S)S 12.1.12, 1.1.1, 22.12.12 (H)H.(M)M 12.23, 7.45, 22.7 (M)M.(S)S 55.58, 1.0