ibm-research/WatsonxDocsQARetrieval huggingface dataset:
# Flow scripting #

You can use scripts to customize operations within a particular flow, and they're saved with that flow\. For example, you might use a script to specify a particular run order for terminal nodes\. You use the flow properties page to edit the script that's saved with the current flow\.

To access scripting in a flow's properties:

<!-- <ol> -->

1.  Right\-click your flow's canvas and select  Flow properties\.
2.  Open the  Scripting section to work with scripts for the current flow\.

<!-- </ol> -->

Tips:

<!-- <ul> -->

 *  By default, the Python scripting language is used\. If you'd rather use a scripting language unique to old versions of  SPSS Modeler desktop, select  Legacy\.
 *  For complete details about scripting, see the [Scripting and automation](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/scripting_guide/clementine/scripting_overview.html) guide\.

<!-- </ul> -->

You can specify whether or not the script runs when the flow runs\. To run the script each time the flow runs, respecting the run order of the script, select  Run the script\. This setting provides automation at the flow level for quicker model building\. Or, to ignore the script, you can select the option to only  Run all terminal nodes when the flow runs\.

The script editor includes the following features that help with script authoring:

<!-- <ul> -->

 *  Syntax highlighting; keywords, literal values (such as strings and numbers), and comments are highlighted
 *  Line numbering
 *  Block matching; when the cursor is placed by the start of a program block, the corresponding end block is also highlighted
 *  Suggested auto\-completion

<!-- </ul> -->

A list of suggested syntax completions can be accessed by selecting  Auto\-Suggest from the context menu, or pressing Ctrl \+ Space\. Use the cursor keys to move up and down the list, then press Enter to insert the selected text\. To exit from auto\-suggest mode without modifying the existing text, press Esc\.

<!-- </article "role="article" "> -->
# Flow scripting example #

You can use a flow to train a model when it runs\. Normally, to test the model, you might run the modeling node to add the model to the flow, make the appropriate connections, and run an Analysis node\.

Using a script, you can automate the process of testing the model nugget after you create it\. For example, you might use a script such as the following to train a neural network model:

    stream = modeler.script.stream()
    neuralnetnode = stream.findByType("neuralnetwork", None)
    results = []
    neuralnetnode.run(results)
    appliernode = stream.createModelApplierAt(results[0], "Drug", 594, 187)
    analysisnode = stream.createAt("analysis", "Drug", 688, 187)
    typenode = stream.findByType("type", None)
    stream.linkBetween(appliernode, typenode, analysisnode)
    analysisnode.run([])

The following bullets describe each line in this script example\.

<!-- <ul> -->

 *  The first line defines a variable that points to the current flow\.
 *  In line 2, the script finds the Neural Net builder node\.
 *  In line 3, the script creates a list where the execution results can be stored\.
 *  In line 4, the Neural Net model nugget is created\. This is stored in the list defined on line 3\.
 *  In line 5, a model apply node is created for the model nugget and placed on the flow canvas\.
 *  In line 6, an analysis node called `Drug` is created\.
 *  In line 7, the script finds the Type node\.
 *  In line 8, the script connects the model apply node created in line 5 between the Type node and the Analysis node\.
 *  Finally, the Analysis node runs to produce the Analysis report\.

<!-- </ul> -->

Tips:

<!-- <ul> -->

 *  It's possible to use a script to build and run a flow from scratch, starting with a blank canvas\.
 *  For complete details about scripting, see the [Scripting and automation](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/scripting_guide/clementine/scripting_overview.html) guide\.

<!-- </ul> -->

<!-- </article "role="article" "> -->
# Mining for concepts and categories #

The Text Mining node uses linguistic and frequency techniques to extract key concepts from the text and create categories with these concepts and other data\. Use the node to explore the text data contents or to produce either a concept model nugget or category model nugget\.

![Text Mining node](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/ta_textmining.png)When you run this node, an internal linguistic extraction engine extracts and organizes the concepts, patterns, and categories by using natural language processing methods\. Two build modes are available in the Text Mining node's properties:

<!-- <ul> -->

 *  The  Generate directly (concept model nugget) mode automatically produces a concept or category model nugget when you run the node\.
 *  The  Build interactively (category model nugget) is a more hands\-on, exploratory approach\. You can use this mode to not only extract concepts, create categories, and refine your linguistic resources, but also run text link analysis and explore clusters\. This build mode launches the Text Analytics Workbench\.

<!-- </ul> -->

And you can use the Text Mining node to generate one of two text mining model nuggets:

<!-- <ul> -->

 *  Concept model nuggets uncover and extract important concepts from your structured or unstructured text data\.
 *  Category model nuggets score and assign documents and records to categories, which are made up of the extracted concepts (and patterns)\.

<!-- </ul> -->

The extracted concepts and patterns and the categories from your model nuggets can all be combined with existing structured data, such as demographics, to yield better and more\-focused decisions\. For example, if customers frequently list login issues as the primary impediment to completing online account management tasks, you might want to incorporate "login issues" into your models\.

<!-- </article "role="article" "> -->
# Nodes palette #

The following sections describe all the nodes available on the palette in SPSS Modeler\. Drag\-and\-drop or double\-click a node in the list to add it to your flow canvas\. You can then double\-click any node icon in your flow to set its properties\. Hover over a property to see information about it, or click the information icon to see Help\.

When first creating a flow, you select which runtime to use\. By default, the flow will use the  IBM SPSS Modeler runtime\. If you want to use native Spark algorithms instead of SPSS algorithms, select the  Spark runtime\. Properties for some nodes will vary depending on which runtime option you choose\.

<!-- </article "role="article" "> -->
# Text Analytics #

SPSS Modeler offers nodes that are specialized for handling text\.

The Text Analytics nodes offer powerful text analytics capabilities, using advanced linguistic technologies and Natural Language Processing (NLP) to rapidly process a large variety of unstructured text data and, from this text, extract and organize the key concepts\. Text Analytics can also group these concepts into categories\.

Around 80% of data held within an organization is in the form of text documents—for example, reports, web pages, e\-mails, and call center notes\. Text is a key factor in enabling an organization to gain a better understanding of their customers' behavior\. A system that incorporates NLP can intelligently extract concepts, including compound phrases\. Moreover, knowledge of the underlying language allows classification of terms into related groups, such as products, organizations, or people, using meaning and context\. As a result, you can quickly determine the relevance of the information to your needs\. These extracted concepts and categories can be combined with existing structured data, such as demographics, and applied to modeling in  SPSS Modeler to yield better and more\-focused decisions\.

Linguistic systems are knowledge sensitive—the more information contained in their dictionaries, the higher the quality of the results\. Text Analytics provides a set of linguistic resources, such as dictionaries for terms and synonyms, libraries, and templates\. These nodes further allow you to develop and refine these linguistic resources to your context\. Fine\-tuning of the linguistic resources is often an iterative process and is necessary for accurate concept retrieval and categorization\. Custom templates, libraries, and dictionaries for specific domains, such as CRM and genomics, are also included\.

Tips for getting started:

<!-- <ul> -->

 *  Watch the following video for an overview of Text Analytics\.
 *  See the [Hotel satisfaction example for Text Analytics](https://dataplatform.cloud.ibm.com/docs/content/wsd/tutorials/tut_ta_hotel.html)\.

<!-- </ul> -->

This video provides a visual method to learn the concepts and tasks in this documentation\.

Video disclaimer: Some minor steps and graphical elements in this video might differ from your platform\.

[https://video\.ibm\.com/embed/channel/23952663/video/spss\-text\-analytics\-workbench](https://video.ibm.com/embed/channel/23952663/video/spss-text-analytics-workbench)

<!-- </article "role="article" "> -->
# About text mining #

Today, an increasing amount of information is being held in unstructured and semi\-structured formats, such as customer e\-mails, call center notes, open\-ended survey responses, news feeds, web forms, etc\. This abundance of information poses a problem to many organizations that ask themselves: How can we collect, explore, and leverage this information?

Text mining is the process of analyzing collections of textual materials in order to capture key concepts and themes and uncover hidden relationships and trends without requiring that you know the precise words or terms that authors have used to express those concepts\. Although they are quite different, text mining is sometimes confused with information retrieval\. While the accurate retrieval and storage of information is an enormous challenge, the extraction and management of quality content, terminology, and relationships contained within the information are crucial and critical processes\.

<!-- </article "role="article" "> -->
# How categorization works #

When creating category models in Text Analytics, there are several different techniques you can choose from to create categories\. Because every dataset is unique, the number of techniques and the order in which you apply them may change\.

Since your interpretation of the results may be different from someone else's, you may need to experiment with the different techniques to see which one produces the best results for your text data\. In Text Analytics, you can create category models in a workbench session in which you can explore and fine\-tune your categories further\.

In this documentation,  category building refers to the generation of category definitions and classification through the use of one or more built\-in techniques, and  categorization refers to the scoring, or labeling, process whereby unique identifiers (name/ID/value) are assigned to the category definitions for each record or document\.

During category building, the concepts and types that were extracted are used as the building blocks for your categories\. When you build categories, the records or documents are automatically assigned to categories if they contain text that matches an element of a category's definition\.

Text Analytics offers you several automated category building techniques to help you categorize your documents or records quickly\.

<!-- </article "role="article" "> -->
# How extraction works #

During the extraction of key concepts and ideas from your responses, Text Analytics relies on linguistics\-based text analysis\. This approach offers the speed and cost effectiveness of statistics\-based systems\. But it offers a far higher degree of accuracy, while requiring far less human intervention\. Linguistics\-based text analysis is based on the field of study known as natural language processing, also known as computational linguistics\.

Understanding how the extraction process works can help you make key decisions when fine\-tuning your linguistic resources (libraries, types, synonyms, and more)\. Steps in the extraction process include:

<!-- <ul> -->

 *  Converting source data to a standard format
 *  Identifying candidate terms
 *  Identifying equivalence classes and integration of synonyms
 *  Assigning a type
 *  Indexing
 *  Matching patterns and events extraction

<!-- </ul> -->

<!-- </article "role="article" "> -->
# Extension nodes #

SPSS Modeler supports the languages R and Apache Spark (via Python)\.

To complement  SPSS Modeler and its data mining abilities, several Extension nodes are available to enable expert users to input their own R scripts or Python for Spark scripts to carry out data processing, model building, and model scoring\.

<!-- <ul> -->

 *  The  Extension Import node is available under  Import on the Node Palette\. See [Extension Import node](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/extension_importer.html)\.
 *  The  Extension Model node is available under  Modeling on the Node Palette\. See [Extension Model node](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/extension_build.html)\.
 *  The  Extension Output node is available under  Outputs on the Node Palette\. See [Extension Output node](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/extension_output.html)\.
 *  The  Extension Export node is available under  Export on the Node Palette\. See [Extension Export node](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/extension_export.html)\.

<!-- </ul> -->

<!-- </article "role="article" "> -->
# Field Operations #

After an initial data exploration, you will probably need to select, clean, or construct data in preparation for analysis\. The Field Operations palette contains many nodes useful for this transformation and preparation\.

For example, using a Derive node, you might create an attribute that is not currently represented in the data\. Or you might use a Binning node to recode field values automatically for targeted analysis\. You will probably find yourself using a Type node frequently—it allows you to assign a measurement level, values, and a modeling role for each field in the dataset\. Its operations are useful for handling missing values and downstream modeling\.

<!-- </article "role="article" "> -->
# Graphs #

Several phases of the data mining process use graphs and charts to explore data brought in to  watsonx\.ai\.

For example, you can connect a Plot or Distribution node to a data source to gain insight into data types and distributions\. You can then perform record and field manipulations to prepare the data for downstream modeling operations\. Another common use of graphs is to check the distribution and relationships between newly derived fields\.

<!-- </article "role="article" "> -->
# Modeling #

Watsonx\.ai offers a variety of modeling methods taken from machine learning, artificial intelligence, and statistics\.

The methods available on the palette allow you to derive new information from your data and to develop predictive models\. Each method has certain strengths and is best suited for particular types of problems\. For more information about modeling, see [Creating SPSS Modeler flows](https://dataplatform.cloud.ibm.com/docs/content/wsd/spss-modeler.html#spss-modeler)\.

<!-- </article "role="article" "> -->
# Record Operations #

Record Operations nodes are useful for making changes to data at the record level\. These operations are important during the data understanding and data preparation phases of data mining because they allow you to tailor the data to your particular business need\.

For example, based on the results of a data audit conducted using the Data Audit node (Outputs palette), you might decide that you would like to merge customer purchase records for the past three months\. Using a Merge node, you can merge records based on the values of a key field, such as `Customer ID`\. Or you might discover that a database containing information about web site hits is unmanageable with over one million records\. Using a Sample node, you can select a subset of data for use in modeling\.

<!-- </article "role="article" "> -->
# Aggregate node #

Aggregation is a data preparation task frequently used to reduce the size of a dataset\. Before proceeding with aggregation, you should take time to clean the data, concentrating especially on missing values\. A aggregation, potentially useful information regarding missing values may be lost\.

You can use an Aggregate node to replace a sequence of input records with summary, aggregated output records\. For example, you might have a set of input sales records such as those shown in the following table\.

<!-- <table "summary="Sales record input example" id="aggregate__table_azf_l25_zcb" class="defaultstyle" "> -->

Sales record input example  

Table 1\. Sales record input example

| Age | Sex | Region | Branch | Sales |
| --- | --- | ------ | ------ | ----- |
| 23  | M   | S      | 8      | 4     |
| 45  | M   | S      | 16     | 4     |
| 37  | M   | S      | 8      | 5     |
| 30  | M   | S      | 5      | 7     |
| 44  | M   | N      | 4      | 9     |
| 25  | M   | N      | 2      | 11    |
| 29  | F   | S      | 16     | 6     |
| 41  | F   | N      | 4      | 8     |
| 23  | F   | N      | 6      | 2     |
| 45  | F   | N      | 4      | 5     |
| 33  | F   | N      | 6      | 10    |

<!-- </table "summary="Sales record input example" id="aggregate__table_azf_l25_zcb" class="defaultstyle" "> -->

You can aggregate these records with `Sex` and `Region` as key fields\. Then choose to aggregate `Age` with the mode  Mean and `Sales` with the mode  Sum\. Select the  Include record count in field aggregate node option and your aggregated output will be similar to the following table\.

<!-- <table "summary="Aggregated record example" id="aggregate__table_ezf_l25_zcb" class="defaultstyle" "> -->

Aggregated record example  

Table 2\. Aggregated record example

| Age (mean) | Sex | Region | Sales (sum) | Record Count |
| ---------- | --- | ------ | ----------- | ------------ |
| 35\.5      | F   | N      | 25          | 4            |
| 29         | F   | S      | 6           | 1            |
| 34\.5      | M   | N      | 20          | 2            |
| 33\.75     | M   | S      | 20          | 4            |

<!-- </table "summary="Aggregated record example" id="aggregate__table_ezf_l25_zcb" class="defaultstyle" "> -->

From this you learn, for example, that the average age of the four female sales staff in the North region is 35\.5, and the sum total of their sales was 25 units\.

Note: Fields such as `Branch` are automatically discarded when no aggregate mode is specified\.
<!-- </article "role="article" "> -->
# Analysis node #

With the Analysis node, you can evaluate the ability of a model to generate accurate predictions\. Analysis nodes perform various comparisons between predicted values and actual values (your target field) for one or more model nuggets\. You can also use Analysis nodes to compare predictive models to other predictive models\.

When you execute an Analysis node, a summary of the analysis results is automatically added to the Analysis section on the Summary tab for each model nugget in the executed flow\. The detailed analysis results appear on the Outputs tab of the manager window or can be written directly to a file\.

Note: Because Analysis nodes compare predicted values to actual values, they are only useful with supervised models (those that require a target field)\. For unsupervised models such as clustering algorithms, there are no actual results available to use as a basis for comparison\.
<!-- </article "role="article" "> -->
# Anonymize node #

With the Anonymize node, you can disguise field names, field values, or both when working with data that's to be included in a model downstream of the node\. In this way, the generated model can be freely distributed (for example, to Technical Support) with no danger that unauthorized users will be able to view confidential data, such as employee records or patients' medical records\.

Depending on where you place the Anonymize node in your flow, you may need to make changes to other nodes\. For example, if you insert an Anonymize node upstream from a Select node, the selection criteria in the Select node will need to be changed if they are acting on values that have now become anonymized\.

The method to be used for anonymizing depends on various factors\. For field names and all field values except Continuous measurement levels, the data is replaced by a string of the form:

    prefix_Sn

where `prefix_` is either a user\-specified string or the default string `anon_`, and `n` is an integer value that starts at 0 and is incremented for each unique value (for example, `anon_S0`, `anon_S1`, etc\.)\.

Field values of type Continuous must be transformed because numeric ranges deal with integer or real values rather than strings\. As such, they can be anonymized only by transforming the range into a different range, thus disguising the original data\. Transformation of a value `x` in the range is performed in the following way:

    A*(x + B)

where:

`A` is a scale factor, which must be greater than 0\.

`B` is a translation offset to be added to the values\.

<!-- </article "role="article" "> -->
# Append node #

You can use Append nodes to concatenate sets of records\. Unlike Merge nodes, which join records from different sources together, Append nodes read and pass downstream all of the records from one source until there are no more\. Then the records from the next source are read using the same data structure (number of records, number of fields, and so on) as the first, or primary, input\. When the primary source has more fields than another input source, the system null string ($null$) will be used for any incomplete values\.

Append nodes are useful for combining datasets with similar structures but different data\. For example, you might have transaction data stored in different files for different time periods, such as a sales data file for March and a separate one for April\. Assuming that they have the same structure (the same fields in the same order), the Append node will join them together into one large file, which you can then analyze\.

Note: To append files, the field measurement levels must be similar\. For example, a `Nominal` field cannot be appended with a field whose measurement level is `Continuous`\.
<!-- </article "role="article" "> -->
# Extension model nugget #

The Extension model nugget is generated and placed on your flow canvas after running the Extension Model node, which contains your R script or Python for Spark script that defines the model building and model scoring\.

By default, the Extension model nugget contains the script that's used for model scoring, options for reading the data, and any output from the R console or Python for Spark\. Optionally, the Extension model nugget can also contain various other forms of model output, such as graphs and text output\. After the Extension model nugget is generated and added to your flow canvas, an output node can be connected to it\. The output node is then used in the usual way within your flow to obtain information about the data and models, and for exporting data in various formats\.

<!-- </article "role="article" "> -->
# Apriori node #

The Apriori node discovers association rules in your data\.

Association rules are statements of the form:

    if  antecedent(s)  then  consequent(s)

For example, if a customer purchases a razor and after shave, then that customer will purchase shaving cream with 80% confidence\. Apriori extracts a set of rules from the data, pulling out the rules with the highest information content\. The Apriori node also discovers association rules in the data\. Apriori offers five different methods of selecting rules and uses a sophisticated indexing scheme to efficiently process large data sets\.

Requirements\. To create an Apriori rule set, you need one or more `Input` fields and one or more `Target` fields\. Input and output fields (those with the role `Input`, `Target`, or `Both`) must be symbolic\. Fields with the role `None` are ignored\. Fields types must be fully instantiated before executing the node\. Data can be in tabular or transactional format\.

Strengths\. For large problems, Apriori is generally faster to train\. It also has no arbitrary limit on the number of rules that can be retained and can handle rules with up to 32 preconditions\. Apriori offers five different training methods, allowing more flexibility in matching the data mining method to the problem at hand\.

<!-- </article "role="article" "> -->
# Auto Cluster node #

The Auto Cluster node estimates and compares clustering models that identify groups of records with similar characteristics\. The node works in the same manner as other automated modeling nodes, enabling you to experiment with multiple combinations of options in a single modeling pass\. Models can be compared using basic measures with which to attempt to filter and rank the usefulness of the cluster models, and provide a measure based on the importance of particular fields\.

Clustering models are often used to identify groups that can be used as inputs in subsequent analyses\. For example, you may want to target groups of customers based on demographic characteristics such as income, or based on the services they have bought in the past\. You can do this without prior knowledge about the groups and their characteristics \-\- you may not know how many groups to look for, or what features to use in defining them\. Clustering models are often referred to as unsupervised learning models, since they do not use a target field, and do not return a specific prediction that can be evaluated as true or false\. The value of a clustering model is determined by its ability to capture interesting groupings in the data and provide useful descriptions of those groupings\.

Requirements\. One or more fields that define characteristics of interest\. Cluster models do not use target fields in the same manner as other models, because they do not make specific predictions that can be assessed as true or false\. Instead, they are used to identify groups of cases that may be related\. For example, you cannot use a cluster model to predict whether a given customer will churn or respond to an offer\. But you can use a cluster model to assign customers to groups based on their tendency to do those things\. Weight and frequency fields are not used\.

Evaluation fields\. While no target is used, you can optionally specify one or more evaluation fields to be used in comparing models\. The usefulness of a cluster model may be evaluated by measuring how well (or badly) the clusters differentiate these fields\.

<!-- </article "role="article" "> -->
# Balance node #

You can use Balance nodes to correct imbalances in datasets so they conform to specified test criteria\.

For example, suppose that a dataset has only two values\-\-`low` or `high`\-\-and that 90% of the cases are `low` while only 10% of the cases are `high`\. Many modeling techniques have trouble with such biased data because they will tend to learn only the *low* outcome and ignore the *high* one, since it is more rare\. If the data is well balanced with approximately equal numbers of `low` and `high` outcomes, models will have a better chance of finding patterns that distinguish the two groups\. In this case, a Balance node is useful for creating a balancing directive that reduces cases with a *low* outcome\.

Balancing is carried out by duplicating and then discarding records based on the conditions you specify\. Records for which no condition holds are always passed through\. Because this process works by duplicating and/or discarding records, the original sequence of your data is lost in downstream operations\. Be sure to derive any sequence\-related values before adding a Balance node to the data stream\.

<!-- </article "role="article" "> -->
# Binning node #

The Binning node enables you to automatically create new nominal fields based on the values of one or more existing continuous (numeric range) fields\. For example, you can transform a continuous income field into a new categorical field containing income groups of equal width, or as deviations from the mean\. Alternatively, you can select a categorical "supervisor" field in order to preserve the strength of the original association between the two fields\.

Binning can be useful for a number of reasons, including:

<!-- <ul> -->

 *  Algorithm requirements\. Certain algorithms, such as Naive Bayes and Logistic Regression, require categorical inputs\.
 *  Performance\. Algorithms such as multinomial logistic may perform better if the number of distinct values of input fields is reduced\. For example, use the median or mean value for each bin rather than using the original values\.
 *  Data Privacy\. Sensitive personal information, such as salaries, may be reported in ranges rather than actual salary figures in order to protect privacy\.

<!-- </ul> -->

A number of binning methods are available\. After you create bins for the new field, you can generate a Derive node based on the cut points\.

<!-- </article "role="article" "> -->
# Caching options for nodes #

To optimize the running of flows, you can set up a cache on any nonterminal node\. When you set up a cache on a node, the cache is filled with the data that passes through the node the next time you run the data flow\. From then on, the data is read from the cache (which is stored temporarily) rather than from the data source\.

Caching is most useful following a time\-consuming operation such as a sort, merge, or aggregation\. For example, suppose that you have an import node set to read sales data from a database and an Aggregate node that summarizes sales by location\. You can set up a cache on the Aggregate node rather than on the import node because you want the cache to store the aggregated data rather than the entire data set\. Note: Caching at import nodes, which simply stores a copy of the original data as it is read into  SPSS Modeler, won't improve performance in most circumstances\.

Nodes with caching enabled are displayed with a special circle\-backslash icon\. When the data is cached at the node, the icon changes to a check mark\.

Figure 1\. Node with empty cache vs\. node with full cache

![Shows a node with an empty cache and a node with a full cache](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/cache_nodes.png)

A circle\-backslash icon by node indicates that its cache is empty\. When the cache is full, the icon becomes a check mark\. If you want to replace the contents of the cache, you must first flush the cache and then re\-run the data flow to refill it\.

In your flow, right\-click the node and select \.

<!-- </article "role="article" "> -->
# CARMA node #

The CARMA node uses an association rules discovery algorithm to discover association rules in the data\.

Association rules are statements in the form:

    if  antecedent(s)  then  consequent(s)

For example, if a Web customer purchases a wireless card and a high\-end wireless router, the customer is also likely to purchase a wireless music server if offered\. The CARMA model extracts a set of rules from the data without requiring you to specify input or target fields\. This means that the rules generated can be used for a wider variety of applications\. For example, you can use rules generated by this node to find a list of products or services (antecedents) whose consequent is the item that you want to promote this holiday season\. Using  watsonx\.ai, you can determine which clients have purchased the antecedent products and construct a marketing campaign designed to promote the consequent product\.

Requirements\. In contrast to Apriori, the CARMA node does not require Input or Target fields\. This is integral to the way the algorithm works and is equivalent to building an Apriori model with all fields set to Both\. You can constrain which items are listed only as antecedents or consequents by filtering the model after it is built\. For example, you can use the model browser to find a list of products or services (antecedents) whose consequent is the item that you want to promote this holiday season\.

To create a CARMA rule set, you need to specify an ID field and one or more content fields\. The ID field can have any role or measurement level\. Fields with the role None are ignored\. Field types must be fully instantiated before executing the node\. Like Apriori, data may be in tabular or transactional format\.

Strengths\. The CARMA node is based on the CARMA association rules algorithm\. In contrast to Apriori, the CARMA node offers build settings for rule support (support for both antecedent and consequent) rather than antecedent support\. CARMA also allows rules with multiple consequents\. Like Apriori, models generated by a CARMA node can be inserted into a data stream to create predictions\.

<!-- </article "role="article" "> -->
# C&R Tree node #

The Classification and Regression (C&R) Tree node is a tree\-based classification and prediction method\. Similar to C5\.0, this method uses recursive partitioning to split the training records into segments with similar output field values\. The C&R Tree node starts by examining the input fields to find the best split, measured by the reduction in an impurity index that results from the split\. The split defines two subgroups, each of which is subsequently split into two more subgroups, and so on, until one of the stopping criteria is triggered\. All splits are binary (only two subgroups)\.

<!-- </article "role="article" "> -->
# CHAID node #

CHAID, or Chi\-squared Automatic Interaction Detection, is a classification method for building decision trees by using chi\-square statistics to identify optimal splits\.

CHAID first examines the crosstabulations between each of the input fields and the outcome, and tests for significance using a chi\-square independence test\. If more than one of these relations is statistically significant, CHAID will select the input field that is the most significant (smallest `p` value)\. If an input has more than two categories, these are compared, and categories that show no differences in the outcome are collapsed together\. This is done by successively joining the pair of categories showing the least significant difference\. This category\-merging process stops when all remaining categories differ at the specified testing level\. For nominal input fields, any categories can be merged; for an ordinal set, only contiguous categories can be merged\.

Exhaustive CHAID is a modification of CHAID that does a more thorough job of examining all possible splits for each predictor but takes longer to compute\.

Requirements\. Target and input fields can be continuous or categorical; nodes can be split into two or more subgroups at each level\. Any ordinal fields used in the model must have numeric storage (not string)\. If necessary, the Reclassify node can be used to convert them\.

Strengths\. Unlike the C&R Tree and QUEST nodes, CHAID can generate nonbinary trees, meaning that some splits have more than two branches\. It therefore tends to create a wider tree than the binary growing methods\. CHAID works for all types of inputs, and it accepts both case weights and frequency variables\.

<!-- </article "role="article" "> -->
# Characters #

Characters (usually shown as `CHAR`) are typically used within a CLEM expression to perform tests on strings\.

For example, you can use the function `isuppercode` to determine whether the first character of a string is uppercase\. The following CLEM expression uses a character to indicate that the test should be performed on the first character of the string:

    isuppercode(subscrs(1, "MyString"))

To express the code (in contrast to the location) of a particular character in a CLEM expression, use single backquotes of the form `` ` ``<*character*>`` ` ``\. For example, `` `A` ``, `` `Z` ``\.

Note: There is no `CHAR` storage type for a field, so if a field is derived or filled with an expression that results in a `CHAR`, then that result will be converted to a string\.
<!-- </article "role="article" "> -->
# CLEM examples #

The example expressions in this section illustrate correct syntax and the types of expressions possible with CLEM\.

Additional examples are discussed throughout this CLEM documentation\. See [CLEM (legacy) language reference](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/clem_reference/clem_language_reference.html#clem_language_reference) for more information\.

<!-- </article "role="article" "> -->
# Functions handling blanks and null values #

Using CLEM, you can specify that certain values in a field are to be regarded as "blanks," or missing values\.

The following functions work with blanks\.

<!-- <table "summary="CLEM blank and null value functions" id="clem_function_ref_blanksnulls__table_shj_4k3_cdb" class="defaultstyle" "> -->

CLEM blank and null value functions  

Table 1\. CLEM blank and null value functions

| Function                 | Result    | Description                                                                                                                                                                                                                                                                                                |
| ------------------------ | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `@BLANK(FIELD)`          | *Boolean* | Returns true for all records whose values are blank according to the blank\-handling rules set in an upstream Type node or Import node (Types tab)\.                                                                                                                                                       |
| `@LAST_NON_BLANK(FIELD)` | *Any*     | Returns the last value for *FIELD* that was not blank, as defined in an upstream Import or Type node\. If there are no nonblank values for *FIELD* in the records read so far, `$null$` is returned\. Note that blank values, also called user\-missing values, can be defined separately for each field\. |
| `@NULL(FIELD)`           | *Boolean* | Returns true if the value of *FIELD* is the system\-missing `$null$.` Returns false for all other values, including user\-defined blanks\. If you want to check for both, use `@BLANK(FIELD)` and`@NULL(FIELD)`\.                                                                                          |
| `undef`                  | *Any*     | Used generally in CLEM to enter a `$null$` value—for example, to fill blank values with nulls in the Filler node\.                                                                                                                                                                                         |

<!-- </table "summary="CLEM blank and null value functions" id="clem_function_ref_blanksnulls__table_shj_4k3_cdb" class="defaultstyle" "> -->

Blank fields may be "filled in" with the Filler node\. In both Filler and Derive nodes (multiple mode only), the special CLEM function `@FIELD` refers to the current field(s) being examined\.

<!-- </article "role="article" "> -->
# Information functions #

You can use information functions to gain insight into the values of a particular field\. They're typically used to derive flag fields\.

For example, the `@BLANK` function creates a flag field indicating records whose values are blank for the selected field\. Similarly, you can check the storage type for a field using any of the storage type functions, such as `is_string`\.

<!-- <table "summary="CLEM information functions" class="defaultstyle" "> -->

CLEM information functions  

Table 1\. CLEM information functions

| Function             | Result    | Description                                                                                                                                          |
| -------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `@BLANK(FIELD)`      | *Boolean* | Returns true for all records whose values are blank according to the blank\-handling rules set in an upstream Type node or source node (Types tab)\. |
| `@NULL(ITEM)`        | *Boolean* | Returns true for all records whose values are undefined\. Undefined values are system null values, displayed in SPSS Modeler as `$null$`\.           |
| `is_date(ITEM)`      | *Boolean* | Returns true for all records whose type is a date\.                                                                                                  |
| `is_datetime(ITEM)`  | *Boolean* | Returns true for all records whose type is a date, time, or timestamp\.                                                                              |
| `is_integer(ITEM)`   | *Boolean* | Returns true for all records whose type is an integer\.                                                                                              |
| `is_number(ITEM)`    | *Boolean* | Returns true for all records whose type is a number\.                                                                                                |
| `is_real(ITEM)`      | *Boolean* | Returns true for all records whose type is a real\.                                                                                                  |
| `is_string(ITEM)`    | *Boolean* | Returns true for all records whose type is a string\.                                                                                                |
| `is_time(ITEM)`      | *Boolean* | Returns true for all records whose type is a time\.                                                                                                  |
| `is_timestamp(ITEM)` | *Boolean* | Returns true for all records whose type is a timestamp\.                                                                                             |

<!-- </table "summary="CLEM information functions" class="defaultstyle" "> -->

<!-- </article "role="article" "> -->
# Probability functions #

Probability functions return probabilities based on various distributions, such as the probability that a value from Student's *t* distribution will be less than a specific value\.

<!-- <table "summary="CLEM probability functions" id="clem_function_ref_probability__table_vqh_xk3_cdb" class="defaultstyle" "> -->

CLEM probability functions  

Table 1\. CLEM probability functions

| Function                        | Result | Description                                                                                                                                               |
| ------------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cdf_chisq(NUM, DF)`            | *Real* | Returns the probability that a value from the chi\-square distribution with the specified degrees of freedom will be less than the specified number\.     |
| `cdf_f(NUM, DF1, DF2)`          | *Real* | Returns the probability that a value from the *F* distribution, with degrees of freedom *DF1* and *DF2*, will be less than the specified number\.         |
| `cdf_normal(NUM, MEAN, STDDEV)` | *Real* | Returns the probability that a value from the normal distribution with the specified mean and standard deviation will be less than the specified number\. |
| `cdf_t(NUM, DF)`                | *Real* | Returns the probability that a value from Student's *t* distribution with the specified degrees of freedom will be less than the specified number\.       |

<!-- </table "summary="CLEM probability functions" id="clem_function_ref_probability__table_vqh_xk3_cdb" class="defaultstyle" "> -->

<!-- </article "role="article" "> -->
# Random functions #

The functions listed on this page can be used to randomly select items or randomly generate numbers\.

<!-- <table "summary="CLEM random functions" id="clem_function_ref_random__table_t2w_sqz_ddb" class="defaultstyle" "> -->

CLEM random functions  

Table 1\. CLEM random functions

| Function       | Result   | Description                                                                                                                                                                                                                                                                                                                                                            |
| -------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `oneof(LIST)`  | *Any*    | Returns a randomly chosen element of *LIST*\. List items should be entered as `[ITEM1,ITEM2,...,ITEM_N]`\. Note that a list of field names can also be specified\.                                                                                                                                                                                                     |
| `random(NUM)`  | *Number* | Returns a uniformly distributed random number of the same type (*INT* or *REAL*), starting from 1 to *NUM*\. If you use an integer, then only integers are returned\. If you use a real (decimal) number, then real numbers are returned (decimal precision determined by the stream options)\. The largest random number returned by the function could equal *NUM*\. |
| `random0(NUM)` | *Number* | This has the same properties as `random(NUM)`, but starting from 0\. The largest random number returned by the function will never equal *NUM*\.                                                                                                                                                                                                                       |

<!-- </table "summary="CLEM random functions" id="clem_function_ref_random__table_t2w_sqz_ddb" class="defaultstyle" "> -->

<!-- </article "role="article" "> -->
# Sequence functions #

For some operations, the sequence of events is important\.

The application allows you to work with the following record sequences:

<!-- <ul> -->

 *  Sequences and time series
 *  Sequence functions
 *  Record indexing
 *  Averaging, summing, and comparing values
 *  Monitoring change—differentiation
 *  `@SINCE`
 *  Offset values
 *  Additional sequence facilities

<!-- </ul> -->

For many applications, each record passing through a stream can be considered as an individual case, independent of all others\. In such situations, the order of records is usually unimportant\.

For some classes of problems, however, the record sequence is very important\. These are typically time series situations, in which the sequence of records represents an ordered sequence of events or occurrences\. Each record represents a snapshot at a particular instant in time; much of the richest information, however, might be contained not in instantaneous values but in the way in which such values are changing and behaving over time\.

Of course, the relevant parameter may be something other than time\. For example, the records could represent analyses performed at distances along a line, but the same principles would apply\.

Sequence and special functions are immediately recognizable by the following characteristics:

<!-- <ul> -->

 *  They are all prefixed by `@`
 *  Their names are given in uppercase

<!-- </ul> -->

Sequence functions can refer to the record currently being processed by a node, the records that have already passed through a node, and even, in one case, records that have yet to pass through a node\. Sequence functions can be mixed freely with other components of CLEM expressions, although some have restrictions on what can be used as their arguments\.

<!-- </article "role="article" "> -->