ibm-research/WatsonxDocsQARetrieval huggingface dataset:
# SoundEx functions #

SoundEx is a method used to find strings when the sound is known but the precise spelling isn't known\.

Developed in 1918, the method searches out words with similar sounds based on phonetic assumptions about how certain letters are pronounced\. SoundEx can be used to search names in a database (for example, where spellings and pronunciations for similar names may vary)\. The basic SoundEx algorithm is documented in a number of sources and, despite known limitations (for example, leading letter combinations such as `ph` and `f` won't match even though they sound the same), is supported in some form by most databases\.

<!-- <table "summary="CLEM soundex functions" id="clem_function_ref_soundex__table_d42_vqz_ddb" class="defaultstyle" "> -->

CLEM soundex functions  

Table 1\. CLEM soundex functions

| Function                               | Result    | Description                                                                                                                                                                                                                        |
| -------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `soundex(STRING)`                      | *Integer* | Returns the four\-character SoundEx code for the specified *STRING*\.                                                                                                                                                              |
| `soundex_difference(STRING1, STRING2)` | *Integer* | Returns an integer between 0 and 4 that indicates the number of characters that are the same in the SoundEx encoding for the two strings, where 0 indicates no similarity and 4 indicates strong similarity or identical strings\. |

<!-- </table "summary="CLEM soundex functions" id="clem_function_ref_soundex__table_d42_vqz_ddb" class="defaultstyle" "> -->

<!-- </article "role="article" "> -->
# Converting date and time values #

Note that conversion functions (and any other functions that require a specific type of input, such as a date or time value) depend on the current formats specified in the flow properties\.

For example, if you have a field named *DATE* that's stored as a string with values *Jan 2021*, *Feb 2021*, and so on, you could convert it to date storage as follows:

    to_date(DATE)

For this conversion to work, select the matching date format  MON YYYY as the default date format for the flow\.

Dates stored as numbers\. Note that `DATE` in the previous example is the name of a field, while `to_date` is a CLEM function\. If you have dates stored as numbers, you can convert them using the `datetime_date` function, where the number is interpreted as a number of seconds since the base date (or epoch)\.

    datetime_date(DATE)

By converting a date to a number of seconds (and back), you can perform calculations such as computing the current date plus or minus a fixed number of days\. For example:

    datetime_date((date_in_days(DATE)-7)*60*60*24)

<!-- </article "role="article" "> -->
# CLEM (legacy) language reference #

This section describes the Control Language for Expression Manipulation (CLEM), which is a powerful tool used to analyze and manipulate the data used in SPSS Modeler flows\.

You can use CLEM within nodes to perform tasks ranging from evaluating conditions or deriving values to inserting data into reports\. CLEM expressions consist of values, field names, operators, and functions\. Using the correct syntax, you can create a wide variety of powerful data operations\.

Figure 1\. Expression Builder

![Expression Builder](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/expressionbuilder_full.png)
<!-- </article "role="article" "> -->
# Working with multiple\-response data #

You can analyze multiple\-response data using a number of comparison functions\.

Available comparison functions include:

<!-- <ul> -->

 *  `value_at`
 *  `first_index / last_index`
 *  `first_non_null / last_non_null`
 *  `first_non_null_index / last_non_null_index`
 *  `min_index / max_index`

<!-- </ul> -->

For example, suppose a multiple\-response question asked for the first, second, and third most important reasons for deciding on a particular purchase (for example, price, personal recommendation, review, local supplier, other)\. In this case, you might determine the importance of price by deriving the index of the field in which it was first included:

    first_index("price", [Reason1 Reason2 Reason3])

Similarly, suppose you asked customers to rank three cars in order of likelihood to purchase and coded the responses in three separate fields, as follows:

<!-- <table "summary="Car ranking example" id="clem_overview_multiple_response_data__table_w55_brz_ddb" class="defaultstyle" "> -->

Car ranking example  

Table 1\. Car ranking example

| customer id | car1 | car2 | car3 |
| ----------- | ---- | ---- | ---- |
| 101         | 1    | 3    | 2    |
| 102         | 3    | 2    | 1    |
| 103         | 2    | 3    | 1    |

<!-- </table "summary="Car ranking example" id="clem_overview_multiple_response_data__table_w55_brz_ddb" class="defaultstyle" "> -->

In this case, you could determine the index of the field for the car they like most (ranked \#1, or the lowest rank) using the `min_index` function:

    min_index(['car1' 'car2' 'car3'])

See [Comparison functions](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/clem_reference/clem_function_ref_comparison.html#clem_function_ref_comparison) for more information\.

<!-- </article "role="article" "> -->
# Working with numbers #

Numerous standard operations on numeric values are available in SPSS Modeler\.

<!-- <ul> -->

 *  Calculating the sine of the specified angle—`sin(NUM)`
 *  Calculating the natural log of numeric fields—`log(NUM)`
 *  Calculating the sum of two numbers—`NUM1` \+ `NUM2`

<!-- </ul> -->

See [Numeric functions](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/clem_reference/clem_function_ref_numeric.html#clem_function_ref_numeric) for more information\.

<!-- </article "role="article" "> -->
# Working with strings #

There are a number of operations available for strings\.

<!-- <ul> -->

 *  Converting a string to uppercase or lowercase—`uppertolower(CHAR)`\.
 *  Removing specified characters, such as `` `ID_` `` or `` `$` ``, from a string variable—`stripchar(CHAR,STRING)`\.
 *  Determining the length (number of characters) for a string variable—`length(STRING).`
 *  Checking the alphabetical ordering of string values—`alphabefore(STRING1, STRING2)`\.
 *  Removing leading or trailing white space from values—`trim(STRING)`, `trim_start(STRING)`, or `trimend(STRING)`\.
 *  Extract the first or last *n* characters from a string—`startstring(LENGTH, STRING)` or `endstring(LENGTH, STRING)`\. For example, suppose you have a field named *item* that combines a product name with a four\-digit ID code (`ACME CAMERA-D109`)\. To create a new field that contains only the four\-digit code, specify the following formula in a Derive node:
    
        endstring(4, item)
 *  Matching a specific pattern—`STRING matches PATTERN`\. For example, to select persons with "market" anywhere in their job title, you could specify the following in a Select node:
    
        job_title matches "*market*"
 *  Replacing all instances of a substring within a string—`replace(SUBSTRING, NEWSUBSTRING, STRING)`\. For example, to replace all instances of an unsupported character, such as a vertical pipe ( `|` ), with a semicolon prior to text mining, use the `replace` function in a Filler node\. Under  Fill in fields in the node properties, select all fields where the character may occur\. For the  Replace condition, select  Always, and specify the following condition under  Replace with\.
    
        replace('|',';',@FIELD)
 *  Deriving a flag field based on the presence of a specific substring\. For example, you could use a string function in a Derive node to generate a separate flag field for each response with an expression such as:

<!-- </ul> -->

    hassubstring(museums,"museum_of_design")

See [String functions](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/clem_reference/clem_function_ref_string.html#clem_function_ref_string) for more information\.

<!-- </article "role="article" "> -->
# Summarizing multiple fields #

The CLEM language includes a number of functions that return summary statistics across multiple fields\.

These functions may be particularly useful in analyzing survey data, where multiple responses to a question may be stored in multiple fields\. See [Working with multiple\-response data](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/clem_reference/clem_overview_multiple_response_data.html#clem_overview_multiple_response_data) for more information\.

<!-- </article "role="article" "> -->
# Handling blanks and missing values #

Replacing blanks or missing values is a common data preparation task for data miners\. CLEM provides you with a number of tools to automate blank handling\.

The Filler node is the most common place to work with blanks; however, the following functions can be used in any node that accepts CLEM expressions:

<!-- <ul> -->

 *  `@BLANK(FIELD)` can be used to determine records whose values are blank for a particular field, such as `Age`\.
 *  `@NULL(FIELD)` can be used to determine records whose values are system\-missing for the specified field(s)\. In SPSS Modeler, system\-missing values are displayed as  $null$ values\.

<!-- </ul> -->

See [Functions handling blanks and null values](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/clem_reference/clem_function_ref_blanksnulls.html#clem_function_ref_blanksnulls) for more information\.

<!-- </article "role="article" "> -->
# Checking CLEM expressions #

Click  Validate in the Expression Builder to validate an expression\.

Expressions that haven't been checked are displayed in red\. If errors are found, a message indicating the cause is displayed\.

The following items are checked:

<!-- <ul> -->

 *  Correct quoting of values and field names
 *  Correct usage of parameters and global variables
 *  Valid usage of operators
 *  Existence of referenced fields
 *  Existence and definition of referenced globals

<!-- </ul> -->

If you encounter errors in syntax, try creating the expression using the lists and operator buttons rather than typing the expression manually\. This method automatically adds the proper quotes for fields and values\.

Note: Field names that contain separators must be surrounded by single quotes\. To automatically add quotes, you can create expressions using the lists and operator buttons rather than typing expressions manually\. The following characters in field names may cause errors: `• ! "# $% & '() = ~ |-^ ¥ @" "+ *" "<>? . ,/ :; →`(arrow mark), `□ △` (graphic mark, etc\.)
<!-- </article "role="article" "> -->
# Selecting functions #

The function list displays all available SPSS Modeler functions and operators\. Scroll to select a function from the list, or, for easier searching, use the drop\-down list to display a subset of functions or operators\. Available functions are grouped into categories for easier searching\.

Most of these categories are described in the Reference section of the CLEM language description\. For more information, see [Functions reference](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/clem_reference/clem_function_ref.html#clem_function_ref)\.

The other categories are as follows\.

<!-- <ul> -->

 *  General Functions\. Contains a selection of some of the most commonly\-used functions\.
 *  Recently Used\. Contains a list of CLEM functions used within the current session\.
 *  @ Functions\. Contains a list of all the special functions, which have their names preceded by an "@" sign\. Note: The `@DIFF1(FIELD1,FIELD2)` and `@DIFF2(FIELD1,FIELD2)` functions require that the two field types are the same (for example, both Integer or both Long or both Real)\.
 *  Database Functions\. If the flow includes a database connection, this selection lists the functions available from within that database, including user\-defined functions (UDFs)\. For more information, see [Database functions](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/clem_reference/expressionbuild_database_functions.html#expressionbuild_database_functions)\.
 *  Database Aggregates\. If the flow includes a database connection, this selection lists the aggregation options available from within that database\. These options are available in the Expression Builder of the Aggregate node\.
 *  Built\-In Aggregates\. Contains a list of the possible modes of aggregation that can be used\.
 *  Operators\. Lists all the operators you can use when building expressions\. Operators are also available from the buttons in the center of the dialog box\.
 *  All Functions\. Contains a complete list of available CLEM functions\.

<!-- </ul> -->

Double\-click a function to insert it into the expression field at the position of the cursor\.

<!-- </article "role="article" "> -->
# The Expression Builder #

You can type CLEM expressions manually or use the Expression Builder, which displays a complete list of CLEM functions and operators as well as data fields from the current flow, allowing you to quickly build expressions without memorizing the exact names of fields or functions\.

In addition, the Expression Builder controls automatically add the proper quotes for fields and values, making it easier to create syntactically correct expressions\.

Notes:

<!-- <ul> -->

 *  The Expression Builder isn't supported in scripting or parameter settings\.
 *  If you want to change your datasource, before changing the source you should check that the Expression Builder can still support the functions you have selected\. Because not all databases support all functions, you may encounter an error if you run against a new datasource\.

<!-- </ul> -->

<!-- </article "role="article" "> -->
# Expressions and conditions #

CLEM expressions can return a result (used when deriving new values)\.

For example:

    Weight * 2.2
    Age + 1
    sqrt(Signal-Echo)

Or, they can evaluate *true* or *false* (used when selecting on a condition)\. For example:

    Drug = "drugA"
    Age < 16
    not(PowerFlux) and Power > 2000

You can combine operators and functions arbitrarily in CLEM expressions\. For example:

    sqrt(abs(Signal))* max(T1, T2) + Baseline

Brackets and operator precedence determine the order in which the expression is evaluated\. In this example, the order of evaluation is:

<!-- <ul> -->

 *  `abs(Signal`) is evaluated, and `sqrt` is applied to its result
 *  `max(T1, T2)` is evaluated
 *  The two results are multiplied:  x has higher precedence than `+`
 *  Finally, `Baseline` is added to the result

<!-- </ul> -->

The descending order of precedence (that is, operations that are performed first to operations that are performed last) is as follows:

<!-- <ul> -->

 *  Function arguments
 *  Function calls
 *  xx
 *  x / mod div rem
 *  `+ –`
 *  `> < >= <= /== == = /=`

<!-- </ul> -->

If you want to override precedence, or if you're in any doubt of the order of evaluation, you can use parentheses to make it explicit\. For example:

    sqrt(abs(Signal))* (max(T1, T2) + Baseline)

<!-- </article "role="article" "> -->
# Viewing or selecting values #

You can view field values from the Expression Builder\. Note that data must be fully instantiated in an Import or Type node to use this feature, so that storage, types, and values are known\.

To view values for a field from the Expression Builder, select the required field and then use the  Value list or perform a search with the  Find in column Value field to find values for the selected field\. You can then double\-click a value to insert it into the current expression or list\.

For flag and nominal fields, all defined values are listed\. For continuous (numeric range) fields, the minimum and maximum values are displayed\.

<!-- </article "role="article" "> -->
# Fields #

Names in CLEM expressions that aren’t names of functions are assumed to be field names\.

You can write these simply as `Power`, `val27`, `state_flag`, and so on, but if the name begins with a digit or includes non\-alphabetic characters, such as spaces (with the exception of the underscore), place the name within single quotation marks (for example, `'Power Increase'`, `'2nd answer'`, `'#101'`, `'$P-NextField'`)\.

Note: Fields that are quoted but undefined in the data set will be misread as strings\.
<!-- </article "role="article" "> -->
# Integers #

Integers are represented as a sequence of decimal digits\.

Optionally, you can place a minus sign (−) before the integer to denote a negative number (for example, `1234`, `999`, −`77`)\.

The CLEM language handles integers of arbitrary precision\. The maximum integer size depends on your platform\. If the values are too large to be displayed in an integer field, changing the field type to `Real` usually restores the value\.

<!-- </article "role="article" "> -->
# About CLEM #

The Control Language for Expression Manipulation (CLEM) is a powerful language for analyzing and manipulating the data that streams through an SPSS Modeler flow\. Data miners use CLEM extensively in flow operations to perform tasks as simple as deriving profit from cost and revenue data or as complex as transforming web log data into a set of fields and records with usable information\.

CLEM is used within SPSS Modeler to:

<!-- <ul> -->

 *  Compare and evaluate conditions on record fields
 *  Derive values for new fields
 *  Derive new values for existing fields
 *  Reason about the sequence of records
 *  Insert data from records into reports

<!-- </ul> -->

CLEM expressions are indispensable for data preparation in SPSS Modeler and can be used in a wide range of nodes—from record and field operations (Select, Balance, Filler) to plots and output (Analysis, Report, Table)\. For example, you can use CLEM in a Derive node to create a new field based on a formula such as ratio\.

CLEM expressions can also be used for global search and replace operations\. For example, the expression `@NULL(@FIELD)` can be used in a Filler node to replace **system\-missing values** with the integer value 0\. (To replace **user\-missing values**, also called blanks, use the `@BLANK` function\.)

More complex CLEM expressions can also be created\. For example, you can derive new fields based on a conditional set of rules, such as a new value category created by using the following expressions: `If: CardID = @OFFSET(CardID,1), Then: @OFFSET(ValueCategory,1), Else: 'exclude'`\.

This example uses the `@OFFSET` function to say: If the value of the field *CardID* for a given record is the same as for the previous record, then return the value of the field named *ValueCategory* for the previous record\. Otherwise, assign the string "exclude\." In other words, if the *CardID*s for adjacent records are the same, they should be assigned the same value category\. (Records with the exclude string can later be culled using a Select node\.)

<!-- </article "role="article" "> -->
# Lists #

A list is an ordered sequence of elements, which may be of mixed type\. Lists are enclosed in square brackets (\[ \])\.

Examples of lists are `[1 2 4 16]` and `["abc" "def"]` and `[A1, A2, A3]`\. Lists are not used as the value of SPSS Modeler fields\. They are used to provide arguments to functions, such as `member` and `oneof`\.

Notes:

<!-- <ul> -->

 *  Lists can be composed only of static objects (for example, a string, number, or field name) and not calls to functions\.
 *  Fields containing a list type aren't supported\. For example, the function `value_at(3, ['Gender' 'BP' 'Cholesterol'])` is supported, but the function `value_at(3, 'ListField')` isn't supported\.

<!-- </ul> -->

<!-- </article "role="article" "> -->
# Reals #

*Real* refers to a floating\-point number\. Reals are represented by one or more digits followed by a decimal point followed by one or more digits\. CLEM reals are held in double precision\.

Optionally, you can place a minus sign (−) before the real to denote a negative number (for example, `1.234`, `0.999`, −`77.001`)\. Use the form <*number*> e <*exponent*> to express a real number in exponential notation (for example, `1234.0e5`, `1.7e`−`2`)\. When SPSS Modeler reads number strings from files and converts them automatically to numbers, numbers with no leading digit before the decimal point or with no digit after the point are accepted (for example, `999.` or `.11`)\. However, these forms are illegal in CLEM expressions\.

Note: When referencing real numbers in CLEM expressions, a period must be used as the decimal separator, regardless of any settings for the current flow or locale\. For example, specify

    Na > 0.6

rather than

    Na > 0,6

This applies even if a comma is selected as the decimal symbol in the flow properties and is consistent with the general guideline that code syntax should be independent of any specific locale or convention\.
<!-- </article "role="article" "> -->
# Strings #

Generally, you should enclose strings in double quotation marks\. Examples of strings are `"c35product2"` and `"referrerID"`\.

To indicate special characters in a string, use a backslash (for example, `"\$65443"`)\. (To indicate a backslash character, use a double backslash, `\\`\.) You can use single quotes around a string, but the result is indistinguishable from a quoted field (`'referrerID'`)\. See [String functions](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/clem_reference/clem_function_ref_string.html#clem_function_ref_string) for more information\.

<!-- </article "role="article" "> -->
# Time #

The CLEM language supports the time formats listed in this section\.

<!-- <table "summary="CLEM language time formats" id="time_formats_clem_language__table_zbw_jyy_ddb" class="defaultstyle" "> -->

CLEM language time formats  

Table 1\. CLEM language time formats

| Format           | Examples                       |
| ---------------- | ------------------------------ |
| `HHMMSS`         | `120112, 010101, 221212`       |
| `HHMM`           | `1223, 0745, 2207`             |
| `MMSS`           | `5558, 0100`                   |
| `HH:MM:SS`       | `12:01:12, 01:01:01, 22:12:12` |
| `HH:MM`          | `12:23, 07:45, 22:07`          |
| `MM:SS`          | `55:58, 01:00`                 |
| `(H)H:(M)M:(S)S` | `12:1:12, 1:1:1, 22:12:12`     |
| `(H)H:(M)M`      | `12:23, 7:45, 22:7`            |
| `(M)M:(S)S`      | `55:58, 1:0`                   |
| `HH.MM.SS`       | `12.01.12, 01.01.01, 22.12.12` |
| `HH.MM`          | `12.23, 07.45, 22.07`          |
| `MM.SS`          | `55.58, 01.00`                 |
| `(H)H.(M)M.(S)S` | `12.1.12, 1.1.1, 22.12.12`     |
| `(H)H.(M)M`      | `12.23, 7.45, 22.7`            |
| `(M)M.(S)S`      | `55.58, 1.0`                   |

<!-- </table "summary="CLEM language time formats" id="time_formats_clem_language__table_zbw_jyy_ddb" class="defaultstyle" "> -->

<!-- </article "role="article" "> -->
# Values and data types #

CLEM expressions are similar to formulas constructed from values, field names, operators, and functions\. The simplest valid CLEM expression is a value or a field name\.

Examples of valid values are:

    3
    1.79
    'banana'

Examples of field names are:

    Product_ID
    '$P-NextField'

where `Product` is the name of a field from a market basket data set, `'$P-NextField'` is the name of a parameter, and the value of the expression is the value of the named field\. Typically, field names start with a letter and may also contain digits and underscores (\_)\. You can use names that don't follow these rules if you place the name within quotation marks\. CLEM values can be any of the following:

<!-- <ul> -->

 *  Strings (for example, `"c1"`, `"Type 2"`, `"a piece of free text"`)
 *  Integers (for example, `12`, `0`, `–189`)
 *  Real numbers (for example, `12.34`, `0.0`, `–0.0045`)
 *  Date/time fields (for example, `05/12/2002`, `12/05/2002`, `12/05/02`)

<!-- </ul> -->

It's also possible to use the following elements:

<!-- <ul> -->

 *  Character codes (for example, `` `a` or 3``)
 *  Lists of items (for example, `[1 2 3]`, `['Type 1' 'Type 2']`)

<!-- </ul> -->

Character codes and lists don't usually occur as field values\. Typically, they're used as arguments of CLEM functions\.

<!-- </article "role="article" "> -->
# Collection node #

Collections are similar to histograms, but collections show the distribution of values for one numeric field relative to the values of another, rather than the occurrence of values for a single field\. A collection is useful for illustrating a variable or field whose values change over time\.

Using 3\-D graphing, you can also include a symbolic axis displaying distributions by category\. Two\-dimensional collections are shown as stacked bar charts, with overlays where used\.

<!-- </article "role="article" "> -->
# Cox node #

Cox Regression builds a predictive model for time\-to\-event data\. The model produces a survival function that predicts the probability that the event of interest has occurred at a given time `t` for given values of the predictor variables\. The shape of the survival function and the regression coefficients for the predictors are estimated from observed subjects; the model can then be applied to new cases that have measurements for the predictor variables\.

Note that information from censored subjects, that is, those that do not experience the event of interest during the time of observation, contributes usefully to the estimation of the model\.

Example\. As part of its efforts to reduce customer churn, a telecommunications company is interested in modeling the time to churn in order to determine the factors that are associated with customers who are quick to switch to another service\. To this end, a random sample of customers is selected, and their time spent as customers (whether or not they are still active customers) and various demographic fields are pulled from the database\.

Requirements\. You need one or more input fields, exactly one target field, and you must specify a survival time field within the Cox node\. The target field should be coded so that the "false" value indicates survival and the "true" value indicates that the event of interest has occurred; it must have a measurement level of `Flag`, with string or integer storage\. (Storage can be converted using a Filler or Derive node if necessary\. ) Fields set to `Both` or `None` are ignored\. Fields used in the model must have their types fully instantiated\. The survival time can be any numeric field\. Note: On scoring a Cox Regression model, an error is reported if empty strings in categorical variables are used as input to model building\. Avoid using empty strings as input\.

Dates & Times\. Date & Time fields cannot be used to directly define the survival time; if you have Date & Time fields, you should use them to create a field containing survival times, based upon the difference between the date of entry into the study and the observation date\.

Kaplan\-Meier Analysis\. Cox regression can be performed with no input fields\. This is equivalent to a Kaplan\-Meier analysis\.

<!-- </article "role="article" "> -->
# CPLEX Optimization node #

With the CPLEX Optimization node, you can use complex mathematical (CPLEX) based optimization via an Optimization Programming Language (OPL) model file\.

For more information about CPLEX optimization and OPL, see the [IBM ILOG CPLEX Optimization Studio documentation](https://www.ibm.com/support/knowledgecenter/SSSA5P)\.

When outputting the data generated by the CPLEX Optimization node, you can output the original data from the data sources together as single indexes, or as multiple dimensional indexes of the result\.

Note:

<!-- <ul> -->

 *  When running a flow containing a CPLEX Optimization node, the CPLEX library has a limitation of 1000 variables and 1000 constraints\.

<!-- </ul> -->

<!-- </article "role="article" "> -->
# Data Asset node #

You can use the Data Asset import node to pull in data from remote data sources using connections or from your local computer\. First, you must create the connection\.

Note for connections to a Planning Analytics database, you must choose a view (not a cube)\.

You can also pull in data from a local data file ( \.csv,  \.txt,  \.json,  \.xls,  \.xlsx,  \.sav, and  \.sas are supported)\. Only the first sheet is imported from spreadsheets\. In the node's properties, under  DATA, select one or more data files to upload\. You can also simply drag\-and\-drop the data file from your local file system onto your canvas\.

Note: You can import a stream ( \.str) into  watsonx\.ai that was created in  SPSS Modeler Subscription or  SPSS Modeler client\. If the imported stream contains one or more import or export nodes, you'll be prompted to convert the nodes\. See [Importing an SPSS Modeler stream](https://dataplatform.cloud.ibm.com/docs/content/wsd/migration.html)\.
<!-- </article "role="article" "> -->
# Data Audit node #

The Data Audit node provides a comprehensive first look at the data you bring to  SPSS Modeler, presented in an interactive, easy\-to\-read matrix that can be sorted and used to generate full\-size graphs\.

When you run a Data Audit node, interactive output is generated that includes:

<!-- <ul> -->

 *  Information such as summary statistics, histograms, box plots, bar charts, pie charts, and more that may be useful in gaining a preliminary understanding of the data\.
 *  Information about outliers, extremes, and missing values\.

<!-- </ul> -->

Figure 1\. Data Audit node output example

![Data Audit node output example](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/spss_data_audit_1.png)

Figure 2\. Data Audit node output example

![Data Audit node output example](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/spss_data_audit_2.png)

Figure 3\. Data Audit node output example

![Data Audit node output example](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/spss_data_audit_3.png)

Figure 4\. Data Audit node output example

![Data Audit node output example](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/spss_data_audit_4.png)

Figure 5\. Data Audit node output example

![Data Audit node output example](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/spss_data_audit_5.png)
<!-- </article "role="article" "> -->
# Decision List node #

Decision List models identify subgroups or segments that show a higher or lower likelihood of a binary (yes or no) outcome relative to the overall sample\.

For example, you might look for customers who are least likely to churn or most likely to say yes to a particular offer or campaign\. The Decision List Viewer gives you complete control over the model, enabling you to edit segments, add your own business rules, specify how each segment is scored, and customize the model in a number of other ways to optimize the proportion of hits across all segments\. As such, it is particularly well\-suited for generating mailing lists or otherwise identifying which records to target for a particular campaign\. You can also use multiple mining tasks to combine modeling approaches—for example, by identifying high\- and low\-performing segments within the same model and including or excluding each in the scoring stage as appropriate\.

<!-- </article "role="article" "> -->
# Derive node #

One of the most powerful features in  watsonx\.ai is the ability to modify data values and derive new fields from existing data\. During lengthy data mining projects, it is common to perform several derivations, such as extracting a customer ID from a string of Web log data or creating a customer lifetime value based on transaction and demographic data\. All of these transformations can be performed, using a variety of field operations nodes\.

Several nodes provide the ability to derive new fields:

<!-- <ul> -->

 *  The  Derive node modifies data values or creates new fields from one or more existing fields\. It creates fields of type formula, flag, nominal, state, count, and conditional\.
 *  The  Reclassify node transforms one set of categorical values to another\. Reclassification is useful for collapsing categories or regrouping data for analysis\.
 *  The  Binning node automatically creates new nominal (set) fields based on the values of one or more existing continuous (numeric range) fields\. For example, you can transform a continuous income field into a new categorical field containing groups of income as deviations from the mean\. After you create bins for the new field, you can generate a Derive node based on the cut points\.
 *  The  Set to Flag node derives multiple flag fields based on the categorical values defined for one or more nominal fields\.
 *  The  Restructure node converts a nominal or flag field into a group of fields that can be populated with the values of yet another field\. For example, given a field named `payment type`, with values of `credit`, `cash`, and `debit`, three new fields would be created (`credit`, `cash`, `debit`), each of which might contain the value of the actual payment made\.

<!-- </ul> -->

Tip: The Control Language for Expression Manipulation (CLEM) is a powerful tool you can use to analyze and manipulate the data used in your flows\. For example, you might use CLEM in a node to derive values\. For more information, see the [CLEM (legacy) language reference](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/clem_reference/clem_language_reference.html)\.
<!-- </article "role="article" "> -->
# Space\-Time\-Boxes node #

Space\-Time\-Boxes (STB) are an extension of Geohashed spatial locations\. More specifically, an STB is an alphanumeric string that represents a regularly shaped region of space and time\.

For example, the STB  dr5ru7\|2013\-01\-01 00:00:00\|2013\-01\-01 00:15:00 is made up of the following three parts:

<!-- <ul> -->

 *  The geohash  dr5ru7
 *  The start timestamp  2013\-01\-01 00:00:00
 *  The end timestamp  2013\-01\-01 00:15:00

<!-- </ul> -->

As an example, you could use space and time information to improve confidence that two entities are the same because they are virtually in the same place at the same time\. Alternatively, you could improve the accuracy of relationship identification by showing that two entities are related due to their proximity in space and time\.

In the node properties, you can choose the  Individual Records or  Hangouts mode as appropriate for your requirements\. Both modes require the same basic details, as follows:

Latitude field\. Select the field that identifies the latitude (in WGS84 coordinate system)\.

Longitude field\. Select the field that identifies the longitude (in WGS84 coordinate system)\.

Timestamp field\. Select the field that identifies the time or date\.

<!-- </article "role="article" "> -->
# Disabling nodes in a flow #

You can disable process nodes that have a single input so that they're ignored when the flow runs\. This saves you from having to remove or bypass the node and means you can leave it connected to the remaining nodes\.

You can still open and edit the node settings; however, any changes will not take effect until you enable the node again\.

For example, you might use a Filter node to filter several fields, and then build models based on the reduced data set\. If you want to also build the same models *without* fields being filtered, to see if they improve the model results, you can disable the Filter node\. When you disable the Filter node, the connections to the modeling nodes pass directly through from the Derive node to the Type node\.

<!-- </article "role="article" "> -->
# Discriminant node #

Discriminant analysis builds a predictive model for group membership\. The model is composed of a discriminant function (or, for more than two groups, a set of discriminant functions) based on linear combinations of the predictor variables that provide the best discrimination between the groups\. The functions are generated from a sample of cases for which group membership is known; the functions can then be applied to new cases that have measurements for the predictor variables but have unknown group membership\.

Example\. A telecommunications company can use discriminant analysis to classify customers into groups based on usage data\. This allows them to score potential customers and target those who are most likely to be in the most valuable groups\.

Requirements\. You need one or more input fields and exactly one target field\. The target must be a categorical field (with a measurement level of `Flag` or `Nominal`) with string or integer storage\. (Storage can be converted using a Filler or Derive node if necessary\. ) Fields set to `Both` or `None` are ignored\. Fields used in the model must have their types fully instantiated\.

Strengths\. Discriminant analysis and Logistic Regression are both suitable classification models\. However, Discriminant analysis makes more assumptions about the input fields—for example, they are normally distributed and should be continuous, and they give better results if those requirements are met, especially if the sample size is small\.

<!-- </article "role="article" "> -->
# Distribution node #

A distribution graph or table shows the occurrence of symbolic (non\-numeric) values, such as mortgage type or gender, in a dataset\. A typical use of the Distribution node is to show imbalances in the data that you can rectify by using a Balance node before creating a model\. You can automatically generate a Balance node using the Generate menu in the distribution graph or table window\.

Note: To show the occurrence of numeric values, you should use a Histogram node\.
<!-- </article "role="article" "> -->
# Charts node #

With the Charts node, you can launch the chart builder and create chart definitions to save with your flow\. Then when you run the node, chart output is generated\.

The Charts node is available under the Graphs section on the node palette\. After adding a Charts node to your flow, double\-click it to open the properties pane\. Then click  Launch Chart Builder to open the chart builder and create one or more chart definitions to associate with the node\. See [Visualizing your data](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/visualizations.html) for details about creating charts\.

Figure 1\. Example charts

![Shows four example charts](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/charts_thumbnail4.png) Notes:

<!-- <ul> -->

 *  When you create a chart, it uses a sample of your data\. After clicking  Save and close to save the chart definition and return to your flow, the Charts node will then use all of your data when you run it\.
 *  Chart definitions are listed in the node properties panel, with icons available for editing them or removing them\.
 *  When you right\-click a Charts node to run it, the defined chart (or charts) is built and added to the Outputs pane\. Open the chart output to interact with it by hovering over it, zooming in or out, or downloading the chart as an image file (\. png)\.
 *  When creating a chart, you can click  Back to flow to close the chart builder and return to your flow\. But you can't run the Charts node until you save a chart definition\.

<!-- </ul> -->

<!-- </article "role="article" "> -->
# Ensemble node #

The Ensemble node combines two or more model nuggets to obtain more accurate predictions than can be gained from any of the individual models\. By combining predictions from multiple models, limitations in individual models may be avoided, resulting in a higher overall accuracy\. Models combined in this manner typically perform at least as well as the best of the individual models and often better\.

This combining of nodes happens automatically in the Auto Classifier and Auto Numeric automated modeling nodes\.

After using an Ensemble node, you can use an Analysis node or Evaluation node to compare the accuracy of the combined results with each of the input models\. To do this, make sure the  Filter out fields generated by ensembled models option is not selected in the Ensemble node settings\.

<!-- </article "role="article" "> -->
# Evaluation node #

The Evaluation node offers an easy way to evaluate and compare predictive models to choose the best model for your application\. Evaluation charts show how models perform in predicting particular outcomes\. They work by sorting records based on the predicted value and confidence of the prediction, splitting the records into groups of equal size (quantiles), and then plotting the value of the business criterion for each quantile, from highest to lowest\. Multiple models are shown as separate lines in the plot\.

Outcomes are handled by defining a specific value or range of values as a hit\. Hits usually indicate success of some sort (such as a sale to a customer) or an event of interest (such as a specific medical diagnosis)\. You can define hit criteria under the  OPTIONS section of the node properties, or you can use the default hit criteria as follows:

<!-- <ul> -->

 *  Flag output fields are straightforward; hits correspond to true values\.
 *  For  Nominal output fields, the first value in the set defines a hit\.
 *  For  Continuous output fields, hits equal values greater than the midpoint of the field's range\.

<!-- </ul> -->

There are six types of evaluation charts, each of which emphasizes a different evaluation criterion\.

Evaluation charts can also be cumulative, so that each point equals the value for the corresponding quantile plus all higher quantiles\. Cumulative charts usually convey the overall performance of models better, whereas noncumulative charts often excel at indicating particular problem areas for models\.

Note: The Evaluation node doesn't support the use of commas in field names\. If you have field names containing commas, you must either remove the commas or surround the field name in quotes\.
<!-- </article "role="article" "> -->
# Expression Builder #

You can type CLEM expressions manually or use the Expression Builder, which displays a complete list of CLEM functions and operators as well as data fields from the current flow, allowing you to quickly build expressions without memorizing the exact names of fields or functions\.

The Expression Builder controls automatically add the proper quotes for fields and values, making it easier to create syntactically correct expressions\.

Notes:

<!-- <ul> -->

 *  The Expression Builder is not supported in parameter settings\.
 *  If you want to change your datasource, before changing the source you should check that the Expression Builder can still support the functions you have selected\. Because not all databases support all functions, you may encounter an error if you run against a new datasource\.
 *  You can run an SPSS Modeler desktop stream file ( \.str) that contains database functions\. But they aren't yet available in the Expression Builder user interface\.

<!-- </ul> -->

Figure 1\. Expression Builder

![Expression Builder](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/expressionbuilder_full.png)
<!-- </article "role="article" "> -->
# Native Python APIs #

You can invoke native Python APIs from your scripts to interact with  SPSS Modeler\.

The following APIs are supported\.

To see an example, you can download the sample stream [python\-extension\-str\.zip](https://github.com/IBMDataScience/ModelerFlowsExamples/blob/main/samples) and import it into  SPSS Modeler (from your project, click  New asset, select  SPSS Modeler, then select  Local file)\. Then open the Extension node properties in the flow to see example syntax\.

<!-- </article "role="article" "> -->
# Extension Transform node #

With the Extension Transform node, you can take data from an  SPSS Modeler flow and apply transformations to the data using R scripting or Python for Spark scripting\.

When the data has been modified, it's returned to the flow for further processing, model building, and model scoring\. The Extension Transform node makes it possible to transform data using algorithms that are written in R or Python for Spark, and enables you to develop data transformation methods that are tailored to a particular problem\.

After adding the node to your canvas, double\-click the node to open its properties\.

<!-- </article "role="article" "> -->
# Python for Spark scripts #

SPSS Modeler supports Python scripts for Apache Spark\.

Note:

<!-- <ul> -->

 *  Python nodes depend on the Spark environment\.
 *  Python scripts must use the Spark API because data is presented in the form of a Spark DataFrame\.
 *  When installing Python, make sure all users have permission to access the Python installation\.
 *  If you want to use the Machine Learning Library (MLlib), you must install a version of Python that includes NumPy\.

<!-- </ul> -->

<!-- </article "role="article" "> -->
# Date, time, timestamp #

For operations that use **date**, **time**, or **timestamp** type data, the value is converted to the real value based on the value `1970-01-01:00:00:00` (using Coordinated Universal Time)\.

For the **date**, the value represents the number of days, based on the value `1970-01-01` (using Coordinated Universal Time)\.

For the **time**, the value represents the number of seconds at 24 hours\.

For the **timestamp**, the value represents the number of seconds based on the value `1970-01-01:00:00:00` (using Coordinated Universal Time)\.

<!-- </article "role="article" "> -->
# PCA/Factor node #

The PCA/Factor node provides powerful data\-reduction techniques to reduce the complexity of your data\. Two similar but distinct approaches are provided\.

<!-- <ul> -->

 *  Principal components analysis (PCA) finds linear combinations of the input fields that do the best job of capturing the variance in the entire set of fields, where the components are orthogonal (perpendicular) to each other\. PCA focuses on all variance, including both shared and unique variance\.
 *  Factor analysis attempts to identify underlying concepts, or factors, that explain the pattern of correlations within a set of observed fields\. Factor analysis focuses on shared variance only\. Variance that is unique to specific fields is not considered in estimating the model\. Several methods of factor analysis are provided by the Factor/PCA node\.

<!-- </ul> -->

For both approaches, the goal is to find a small number of derived fields that effectively summarize the information in the original set of fields\.

Requirements\. Only numeric fields can be used in a PCA\-Factor model\. To estimate a factor analysis or PCA, you need one or more fields with the role set to `Input` fields\. Fields with the role set to `Target`, `Both`, or `None` are ignored, as are non\-numeric fields\.

Strengths\. Factor analysis and PCA can effectively reduce the complexity of your data without sacrificing much of the information content\. These techniques can help you build more robust models that execute more quickly than would be possible with the raw input fields\.

<!-- </article "role="article" "> -->