itsjhuang/watsonx-docs-document-type huggingface dataset:
Selecting functions (SPSS Modeler)
Selecting functions The function list displays all available SPSS Modeler functions and operators. Scroll to select a function from the list, or, for easier searching, use the drop-down list to display a subset of functions or operators. Available functions are grouped into categories for easier searching. Most of these categories are described in the Reference section of the CLEM language description. For more information, see [Functions reference](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/clem_reference/clem_function_ref.htmlclem_function_ref). The other categories are as follows. * General Functions. Contains a selection of some of the most commonly-used functions. * Recently Used. Contains a list of CLEM functions used within the current session. * @ Functions. Contains a list of all the special functions, which have their names preceded by an "@" sign. Note: The @DIFF1(FIELD1,FIELD2) and @DIFF2(FIELD1,FIELD2) functions require that the two field types are the same (for example, both Integer or both Long or both Real). * Database Functions. If the flow includes a database connection, this selection lists the functions available from within that database, including user-defined functions (UDFs). For more information, see [Database functions](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/clem_reference/expressionbuild_database_functions.htmlexpressionbuild_database_functions). * Database Aggregates. If the flow includes a database connection, this selection lists the aggregation options available from within that database. These options are available in the Expression Builder of the Aggregate node. * Built-In Aggregates. Contains a list of the possible modes of aggregation that can be used. * Operators. Lists all the operators you can use when building expressions. Operators are also available from the buttons in the center of the dialog box. * All Functions. Contains a complete list of available CLEM functions. Double-click a function to insert it into the expression field at the position of the cursor.
About text mining (SPSS Modeler)
About text mining Today, an increasing amount of information is being held in unstructured and semi-structured formats, such as customer e-mails, call center notes, open-ended survey responses, news feeds, web forms, etc. This abundance of information poses a problem to many organizations that ask themselves: How can we collect, explore, and leverage this information? Text mining is the process of analyzing collections of textual materials in order to capture key concepts and themes and uncover hidden relationships and trends without requiring that you know the precise words or terms that authors have used to express those concepts. Although they are quite different, text mining is sometimes confused with information retrieval. While the accurate retrieval and storage of information is an enormous challenge, the extraction and management of quality content, terminology, and relationships contained within the information are crucial and critical processes.
Set Globals node (SPSS Modeler)
Set Globals node The Set Globals node scans the data and computes summary values that can be used in CLEM expressions. For example, you can use a Set Globals node to compute statistics for a field called age and then use the overall mean of age in CLEM expressions by inserting the function @GLOBAL_MEAN(age).
Troubleshooting information for SPSS Modeler
Troubleshooting SPSS Modeler The information in this section provides troubleshooting details for issues you may encounter in SPSS Modeler.
ensemblenode properties
ensemblenode properties ![Ensemble node icon](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/scripting_guide/clementine/images/ensemblenodeicon.png)The Ensemble node combines two or more model nuggets to obtain more accurate predictions than can be gained from any one model. ensemblenode properties Table 1. ensemblenode properties ensemblenode properties Data type Property description ensemble_target_field field Specifies the target field for all models used in the ensemble. filter_individual_model_output flag Specifies whether scoring results from individual models should be suppressed. flag_ensemble_method VotingConfidenceWeightedVotingRawPropensityWeightedVotingAdjustedPropensityWeightedVotingHighestConfidenceAverageRawPropensityAverageAdjustedPropensity Specifies the method used to determine the ensemble score. This setting applies only if the selected target is a flag field. set_ensemble_method VotingConfidenceWeightedVotingHighestConfidence Specifies the method used to determine the ensemble score. This setting applies only if the selected target is a nominal field. flag_voting_tie_selection RandomHighestConfidenceRawPropensityAdjustedPropensity If a voting method is selected, specifies how ties are resolved. This setting applies only if the selected target is a flag field. set_voting_tie_selection RandomHighestConfidence If a voting method is selected, specifies how ties are resolved. This setting applies only if the selected target is a nominal field. calculate_standard_error flag If the target field is continuous, a standard error calculation is run by default to calculate the difference between the measured or estimated values and the true values; and to show how close those estimates matched.
Gaussian Mixture node (SPSS Modeler)
Gaussian Mixture node A Gaussian Mixture© model is a probabilistic model that assumes all the data points are generated from a mixture of a finite number of Gaussian distributions with unknown parameters. One can think of mixture models as generalizing k-means clustering to incorporate information about the covariance structure of the data as well as the centers of the latent Gaussians.^1^ The Gaussian Mixture node in watsonx.ai exposes the core features and commonly used parameters of the Gaussian Mixture library. The node is implemented in Python. For more information about Gaussian Mixture modeling algorithms and parameters, see [Gaussian Mixture Models](http://scikit-learn.org/stable/modules/mixture.html) and [Gaussian Mixture](https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html). ^2^ ^1^ [User Guide.](https://scikit-learn.org/stable/modules/mixture.html)Gaussian mixture models. Web. © 2007 - 2017. scikit-learn developers. ^2^ [Scikit-learn: Machine Learning in Python](http://jmlr.csail.mit.edu/papers/v12/pedregosa11a.html), Pedregosa et al., JMLR 12, pp. 2825-2830, 2011.
Creating the flow (SPSS Modeler)
Creating the flow 1. Create a new flow and add a Data Asset node that points to catalog_seasfac.csv. 2. Connect a Type node to the Data Asset node and double-click it to open its properties. 3. Click Read Values. For the men field, set the role to Target. Figure 1. Specifying the target field ![Specifying the target field](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/tut_forecast_fields.png) 4. Set the role for all other fields to None and click Save. 5. Attach a Time Plot graph node to the Type node and double-click it. Figure 2. Plotting the time series ![Plotting the time series](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/tut_forecast_plot.png) 6. For the Plot, add the field men to the Series list. 7. Select Use custom x axis field label and select date. 8. Deselect the Normalize option and click Save. 9. Run the flow.
evaluationnode properties
evaluationnode properties ![Evaluation node icon](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/scripting_guide/clementine/images/evaluationnodeicon.png)The Evaluation node helps to evaluate and compare predictive models. The evaluation chart shows how well models predict particular outcomes. It sorts records based on the predicted value and confidence of the prediction. It splits the records into groups of equal size (quantiles) and then plots the value of the business criterion for each quantile from highest to lowest. Multiple models are shown as separate lines in the plot. evaluationnode properties Table 1. evaluationnode properties evaluationnode properties Data type Property description chart_type Gains <br>Response <br>Lift <br>Profit <br>ROI <br>ROC inc_baseline flag field_detection_method Metadata <br>Name use_fixed_cost flag cost_value number cost_field string use_fixed_revenue flag revenue_value number revenue_field string use_fixed_weight flag weight_value number weight_field field n_tile Quartiles <br>Quintles <br>Deciles <br>Vingtiles <br>Percentiles <br>1000-tiles cumulative flag style Line <br>Point point_type Rectangle <br>Dot <br>Triangle <br>Hexagon <br>Plus <br>Pentagon <br>Star <br>BowTie <br>HorizontalDash <br>VerticalDash <br>IronCross <br>Factory <br>House <br>Cathedral <br>OnionDome <br>ConcaveTriangleOblateGlobe <br>CatEye <br>FourSidedPillow <br>RoundRectangle <br>Fan export_data flag data_filename string delimiter string new_line flag inc_field_names flag inc_best_line flag inc_business_rule flag business_rule_condition string plot_score_fields flag score_fields [field1 ... fieldN] target_field field use_hit_condition flag hit_condition string use_score_expression flag score_expression string caption_auto flag split_by_partition boolean If a partition field is used to split records into training, test, and validation samples, use this option to display a separate evaluation chart for each partition. use_profit_criteria boolean Enables profit criteria. use_grid boolean Displays grid lines.
Tokens and tokenization
Tokens and tokenization A token is a collection of characters that has semantic meaning for a model. Tokenization is the process of converting the words in your prompt into tokens. You can monitor foundation model token usage in a project on the Environments page on the Resource usage tab. Converting words to tokens and back again Prompt text is converted to tokens before being processed by foundation models. The correlation between words and tokens is complex: * Sometimes a single word is broken into multiple tokens * The same word might be broken into a different number of tokens, depending on context (such as: where the word appears, or surrounding words) * Spaces, newline characters, and punctuation are sometimes included in tokens and sometimes not * The way words are broken into tokens varies from language to language * The way words are broken into tokens varies from model to model For a rough idea, a sentence that has 10 words could be 15 to 20 tokens. The raw output from a model is also tokens. In the Prompt Lab in IBM watsonx.ai, the output tokens from the model are converted to words to be displayed in the prompt editor. Example The following image shows how this sample input might be tokenized: > Tomatoes are one of the most popular plants for vegetable gardens. Tip for success: If you select varieties that are resistant to disease and pests, growing tomatoes can be quite easy. For experienced gardeners looking for a challenge, there are endless heirloom and specialty varieties to cultivate. Tomato plants come in a range of sizes. ![Visualization of tokenization](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/images/fm-tokenization.png) Notice a few interesting points: * Some words are broken into multiple tokens and some are not * The word "Tomatoes" is broken into multiple tokens at the beginning, but later "tomatoes" is all one token * Spaces are sometimes included at the beginning of a word-token and sometimes spaces are a token all by themselves * Punctuation marks are tokens Token limits Every model has an upper limit to the number of tokens in the input prompt plus the number of tokens in the generated output from the model (sometimes called context window length, context window, context length, or maximum sequence length.) In the Prompt Lab, an informational message shows how many tokens are used in a given prompt submission and the resulting generated output. In the Prompt Lab, you use the Max tokens parameter to specify an upper limit on the number of output tokens for the model to generate. The maximum number of tokens that are allowed in the output differs by model. For more information, see the Maximum tokens information in [Supported foundation models](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models.html). Parent topic:[Foundation models](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-overview.html)
Extension Transform node (SPSS Modeler)
Extension Transform node With the Extension Transform node, you can take data from an SPSS Modeler flow and apply transformations to the data using R scripting or Python for Spark scripting. When the data has been modified, it's returned to the flow for further processing, model building, and model scoring. The Extension Transform node makes it possible to transform data using algorithms that are written in R or Python for Spark, and enables you to develop data transformation methods that are tailored to a particular problem. After adding the node to your canvas, double-click the node to open its properties.
Building a chart from the chart type gallery
Building a chart from the chart type gallery Use chart type gallery for building charts. Following are general steps for building a chart from the gallery. 1. In the Chart Type section, select a chart category. A preview version of the selected chart type is shown on the chart canvas. 2. If the canvas already displays a chart, the new chart replaces the chart's axis set and graphic elements. 1. Depending on the selected chart type, the available variables are presented under a number of different headings in the Details pane (for example, Category for bar charts, X-axis and Y-axis for line charts). Select the appropriate variables for the selected chart type. 3. Click the Save visualization to project control to save the visualization to the project. You can select to also Create a new asset from the visualization, provide a visualization asset name, description, and chart name. 4. Click Apply to save the visualization to the project. The new visualization asset is now available under the Assets tab.
Collaboration roles for governance
Collaboration roles for governance Review the collaboration roles for managing access to governance tools such as inventories, AI use cases, and evaluations. User roles and permissions for governance The permissions that you allow you to work with governance artifacts depend on your watsonx roles: * IAM Platform access roles determine your permissions for the IBM Cloud account. At least the Viewer role is required to work with services. * IAM Service access roles determine your permissions within services. * Workspace collaborator roles determine what actions you have permission to perform within workspaces in IBM watsonx. For details, see [Levels of user access roles in IBM watsonx](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/roles.html). Roles for governance If you have the IAM Platform Admin role, you can: * Provision watsonx.governance * Create inventory * Create platform assets catalog * Enable external model tracking * Create attachment fact definitions * Customize report templates If you have these workspace roles for an inventory, you can: Governance permissions for inventories Enabled permission Viewer Editor Admin/Owner Create and edit AI use cases ✓ ✓ View AI use cases ✓ ✓ ✓ Add collaborators to an inventory ✓ Delete inventory ✓ Evaluate model deployment ✓ ✓ Add collaborators to a use case ✓ ✓ Generate reports ✓ ✓ ✓ Add attachments to a use case ✓ ✓ Update asset type definitions <br>(For example: model_entry_user, modelfacts_user) ✓ If you have these workspace roles for an AI use case, you can: Governance permissions for AI use cases Enabled permission Editor/Collaborator Admin/Owner Delete AI use cases ✓ Add collaborators to the use case ✓ Edit AI use case ✓ ✓ Edit use case ✓ ✓ Add values to custom facts ✓ ✓ Upload attachments to use case ✓ ✓ If you have these workspace roles for a project or space, you can: Governance permissions for project and space roles Enabled permission Viewer Editor/Collaborator Admin/Owner Track/untrack prompt template ✓ ✓ Upload attachments to use case ✓ ✓ Add values to custom facts ✓ ✓ View AI factsheet ✓ ✓ ✓ Generate report ✓ ✓ ✓ Learn more Parent topic:[Governing assets in AI use cases](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/xgov-use-cases.html)
Types of scripts
Types of scripts SPSS Modeler uses three types of scripts: * Flow scripts are stored as a flow property and are therefore saved and loaded with a specific flow. For example, you can write a flow script that automates the process of training and applying a model nugget. You can also specify that whenever a particular flow runs, the script should be run instead of the flow's canvas content. * Standalone scripts aren't associated with any particular flow and are saved in external text files. You might use a standalone script, for example, to manipulate multiple flows together. * SuperNode scripts are stored as a SuperNode flow property. SuperNode scripts are only available in terminal SuperNodes. You might use a SuperNode script to control the execution sequence of the SuperNode contents. For nonterminal (import or process) SuperNodes, you can define properties for the SuperNode or the nodes it contains in your flow script directly.
Strings
Strings A string is an immutable sequence of characters that's treated as a value. Strings support all of the immutable sequence functions and operators that result in a new string. For example, "abcdef"[1:4] results in the output "bcd". In Python, characters are represented by strings of length one. Strings literals are defined by the use of single or triple quoting. Strings that are defined using single quotes can't span lines, while strings that are defined using triple quotes can. You can enclose a string in single quotes (') or double quotes ("). A quoting character may contain the other quoting character un-escaped or the quoting character escaped, that's preceded by the backslash () character.
Working with strings (SPSS Modeler)
Working with strings There are a number of operations available for strings. * Converting a string to uppercase or lowercase—uppertolower(CHAR). * Removing specified characters, such as ID_ or $ , from a string variable—stripchar(CHAR,STRING). * Determining the length (number of characters) for a string variable—length(STRING). * Checking the alphabetical ordering of string values—alphabefore(STRING1, STRING2). * Removing leading or trailing white space from values—trim(STRING), trim_start(STRING), or trimend(STRING). * Extract the first or last n characters from a string—startstring(LENGTH, STRING) or endstring(LENGTH, STRING). For example, suppose you have a field named item that combines a product name with a four-digit ID code (ACME CAMERA-D109). To create a new field that contains only the four-digit code, specify the following formula in a Derive node: endstring(4, item) * Matching a specific pattern—STRING matches PATTERN. For example, to select persons with "market" anywhere in their job title, you could specify the following in a Select node: job_title matches "market" * Replacing all instances of a substring within a string—replace(SUBSTRING, NEWSUBSTRING, STRING). For example, to replace all instances of an unsupported character, such as a vertical pipe ( | ), with a semicolon prior to text mining, use the replace function in a Filler node. Under Fill in fields in the node properties, select all fields where the character may occur. For the Replace condition, select Always, and specify the following condition under Replace with. replace('|',';',@FIELD) * Deriving a flag field based on the presence of a specific substring. For example, you could use a string function in a Derive node to generate a separate flag field for each response with an expression such as: hassubstring(museums,"museum_of_design") See [String functions](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/clem_reference/clem_function_ref_string.htmlclem_function_ref_string) for more information.
History node (SPSS Modeler)
History node History nodes are most often used for sequential data, such as time series data. They are used to create new fields containing data from fields in previous records. When using a History node, you may want to use data that is presorted by a particular field. You can use a Sort node to do this.
applytreeas properties
applytreeas properties You can use Tree-AS modeling nodes to generate a Tree-AS model nugget. The scripting name of this model nugget is applytreenas. For more information on scripting the modeling node itself, see [treeas properties](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/scripting_guide/clementine/treeASnodeslots.htmltreeASnodeslots). applytreeas properties Table 1. applytreeas properties applytreeas Properties Values Property description calculate_conf flag This property includes confidence calculations in the generated tree. display_rule_id flag Adds a field in the scoring output that indicates the ID for the terminal node to which each record is assigned. enable_sql_generation falsenative When using data from a database, SQL code can be pushed back to the database for execution, providing superior performance for many operations.
anonymizenode properties
anonymizenode properties ![Anonymize node icon](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/scripting_guide/clementine/images/anonymizenodeicon.png)The Anonymize node transforms the way field names and values are represented downstream, thus disguising the original data. This can be useful if you want to allow other users to build models using sensitive data, such as customer names or other details. anonymizenode properties Table 1. anonymizenode properties anonymizenode properties Data type Property description enable_anonymize flag When set to True, activates anonymization of field values (equivalent to selecting Yes for that field in the Anonymize Values column). use_prefix flag When set to True, a custom prefix will be used if one has been specified. Applies to fields that will be anonymized by the Hash method and is equivalent to choosing the Custom option in the Replace Values settings for that field. prefix string Equivalent to typing a prefix into the text box in the Replace Values settings. The default prefix is the default value if nothing else has been specified. transformation RandomFixed Determines whether the transformation parameters for a field anonymized by the Transform method will be random or fixed. set_random_seed flag When set to True, the specified seed value will be used (if transformation is also set to Random). random_seed integer When set_random_seed is set to True, this is the seed for the random number. scale number When transformation is set to Fixed, this value is used for "scale by." The maximum scale value is normally 10 but may be reduced to avoid overflow. translate number When transformation is set to Fixed, this value is used for "translate." The maximum translate value is normally 1000 but may be reduced to avoid overflow.
Marking a project as sensitive
Marking a project as sensitive When you create a project, you can mark the project as sensitive to prevent project collaborators from moving sensitive data out of the project. Marking a project as sensitive prevents collaborators of a project, including administrators, from downloading or exporting data assets, connections, or connected data from a project. These sensitive assets cannot be added to a catalog or promoted to a space either. Project collaborators with Admin or Editor role can export assets like notebooks or models from the project. When users open a project that is marked as sensitive, a notification is displayed stating that no data assets can be downloaded or exported from the project. Restrictions * You cannot mark a project as sensitive after the project is created. * You cannot mark projects that use Git integration as sensitive. Parent topic:[Administering projects](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/admin-project.html)