itsjhuang/watsonx-docs-document-type huggingface dataset:
Analyzing data and working with models
Analyzing data and working with models You can analyze data and build or work with models in projects. The methods that you choose for preparing data or working models help you determine which tools best fit your needs. Each tool has a specific, primary task. Some tools have capabilities for multiple types of tasks. You can choose a tool based on how much automation you want: * Code editor tools: Use to write code in Python or R, all also with Spark. * Graphical builder tools: Use menus and drag-and-drop functionality on a builder to visually program. * Automated builder tools: Use to configure automated tasks that require limited user input. Tool to tasks Tool Primary task Tool type Work with data Work with models [Data Refinery](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/refining_data.html) Prepare and visualize data Graphical builder ✓ [Visualizations](https://dataplatform.cloud.ibm.com/docs/content/dataview/idh_idc_cg_help_main.html) Build graphs to visualize data Graphical builder ✓ [Prompt Lab](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-prompt-lab.html) Experiment with foundation models and prompts Graphical builder ✓ [Tuning Studio](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-tuning-studio.html) Tune a foundation model to return output in a certain style or format Graphical builder ✓ ✓ [Jupyter notebook editor](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/notebook-editor.html) Work with data and models in Python or R notebooks Code editor ✓ ✓ [Federated learning](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fed-lea.html) Train models on distributed data Code editor ✓ [RStudio IDE](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/rstudio-overview.html) Work with data and models in R Code editor ✓ ✓ [SPSS Modeler](https://dataplatform.cloud.ibm.com/docs/content/wsd/spss-modeler.html) Build models as a visual flow Graphical builder ✓ ✓ [Decision Optimization](https://dataplatform.cloud.ibm.com/docs/content/DO/DOWS-Cloud_home.html) Solve optimization problems Graphical builder, code editor ✓ ✓ [AutoAI tool](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/autoai-overview.html) Build machine learning models automatically Automated builder ✓ ✓ [Pipelines](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-orchestration-overview.html) Automate model lifecycle Graphical builder ✓ ✓ [Synthetic Data Generator](https://dataplatform.cloud.ibm.com/docs/content/wsj/synthetic/synthetic_data_overview_sd.html) Generate synthetic tabular data Graphical builder ✓ ✓ Learn more * [Foundation models](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-overview.html) * [Notebooks and scripts](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/notebooks-and-scripts.html)
{{ document.title.text }}
Data bias ![icon for fairness risk](https://dataplatform.cloud.ibm.com/docs/content/wsj/ai-risk-atlas/images/risk-fairness.svg)Risks associated with inputTraining and tuning phaseFairnessAmplified Description Historical, representational, and societal biases present in the data used to train and fine tune the model can adversely affect model behavior. Why is data bias a concern for foundation models? Training an AI system on data with bias, such as historical or representational bias, could lead to biased or skewed outputs that may unfairly represent or otherwise discriminate against certain groups or individuals. In addition to negative societal impacts, business entities could face legal consequences or reputational harms from biased model outcomes. Example Healthcare Bias Research on reinforcing disparities in medicine highlights that using data and AI to transform how people receive healthcare is only as strong as the data behind it, meaning use of training data with poor minority representation can lead to growing health inequalities. Sources: [Science, September 2022](https://www.science.org/doi/10.1126/science.abo2788) [Forbes, December 2022](https://www.forbes.com/sites/adigaskell/2022/12/02/minority-patients-often-left-behind-by-health-ai/?sh=31d28a225b41) Parent topic:[AI risk atlas](https://dataplatform.cloud.ibm.com/docs/content/wsj/ai-risk-atlas/ai-risk-atlas.html)
Append node (SPSS Modeler)
Append node You can use Append nodes to concatenate sets of records. Unlike Merge nodes, which join records from different sources together, Append nodes read and pass downstream all of the records from one source until there are no more. Then the records from the next source are read using the same data structure (number of records, number of fields, and so on) as the first, or primary, input. When the primary source has more fields than another input source, the system null string ($null$) will be used for any incomplete values. Append nodes are useful for combining datasets with similar structures but different data. For example, you might have transaction data stored in different files for different time periods, such as a sales data file for March and a separate one for April. Assuming that they have the same structure (the same fields in the same order), the Append node will join them together into one large file, which you can then analyze. Note: To append files, the field measurement levels must be similar. For example, a Nominal field cannot be appended with a field whose measurement level is Continuous.
Disabling or caching nodes in a flow (SPSS Modeler)
Disabling or caching nodes in a flow You can disable a node so it's ignored when the flow runs. And you can set up a cache on a node.
{{ document.title.text }}
Personal information in data ![icon for privacy risk](https://dataplatform.cloud.ibm.com/docs/content/wsj/ai-risk-atlas/images/risk-privacy.svg)Risks associated with inputTraining and tuning phasePrivacyTraditional Description Inclusion or presence of personal identifiable information (PII) and sensitive personal information (SPI) in the data used for training or fine tuning the model might result in unwanted disclosure of that information. Why is personal information in data a concern for foundation models? If not properly developed to protect sensitive data, the model might expose personal information in the generated output. Additionally, personal or sensitive data must be reviewed and handled with respect to privacy laws and regulations, as business entities could face fines, reputational harms, and other legal consequences if found in violation. Example Training on Private Information According to the article, Google and its parent company Alphabet were accused in a class-action lawsuit of misusing vast amount of personal information and copyrighted material taken from what is described as hundreds of millions of internet users to train its commercial AI products, which includes Bard, its conversational generative artificial intelligence chatbot. This follows similar lawsuits filed against Meta Platforms, Microsoft, and OpenAI over their alleged misuse of personal data. Sources: [Reuters, July 2023](https://www.reuters.com/legal/litigation/google-hit-with-class-action-lawsuit-over-ai-data-scraping-2023-07-11/) [J.L. v. Alphabet Inc., July 2023](https://fingfx.thomsonreuters.com/gfx/legaldocs/myvmodloqvr/GOOGLE%20AI%20LAWSUIT%20complaint.pdf) Parent topic:[AI risk atlas](https://dataplatform.cloud.ibm.com/docs/content/wsj/ai-risk-atlas/ai-risk-atlas.html)
Building the flow (SPSS Modeler)
Building the flow Figure 1. Feature Selection example flow ![Feature Selection example flow](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/tut_screening.png) 1. Add a Data Asset node that points to customer_dbase.csv. 2. Add a Type node after the Data Asset node. 3. Double-click the Type node to open its properties, and change the role for response_01 to Target. Change the role to None for the other response fields (response_02 and response_03) and for the customer ID (custid) field. Leave the role set to Input for all other fields. Figure 2. Adding a Type node ![Adding a Type node](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/tut_screening_target.png) 4. Click Read Values and then click Save. 5. Add a Feature Selection modeling node after the Type node. In the node properties, the rules and criteria used for screening or disqualifying fields are defined. Figure 3. Adding a Feature Selection node ![Adding a Feature Selection node](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/tut_screening_criteria.png) 6. Run the flow to generate the Feature Selection model nugget. 7. To look at the results, right-click the model nugget and choose View Model. The results show the fields found to be useful in the prediction, ranked by importance. By examining these fields, you can decide which ones to use in subsequent modeling sessions. 8. To compare results without feature selection, you must add two CHAID modeling nodes to the flow: one that uses feature selection and one that doesn't. Add two CHAID nodes, one connected to the Type node and the other connected to the Feature Selection model nugget, as shown in the example flow at the beginning of this section. 9. Double-click each CHAID node to open its properties. Under Objectives, make sure that Build new model and Create a standard model are selected. Under , select Custom and set it to 5.
Example: Searching for nodes using a custom filter
Example: Searching for nodes using a custom filter The section [Finding nodes](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/scripting_guide/clementine/jython/clementine/python_node_find.htmlpython_node_find) includes an example of searching for a node in a flow using the type name of the node as the search criterion. In some situations, a more generic search is required and this can be accomplished by using the NodeFilter class and the flow findAll() method. This type of search involves the following two steps: 1. Creating a new class that extends NodeFilter and that implements a custom version of the accept() method. 2. Calling the flow findAll() method with an instance of this new class. This returns all nodes that meet the criteria defined in the accept() method. The following example shows how to search for nodes in a flow that have the node cache enabled. The returned list of nodes can be used to either flush or disable the caches of these nodes. import modeler.api class CacheFilter(modeler.api.NodeFilter): """A node filter for nodes with caching enabled""" def accept(this, node): return node.isCacheEnabled() cachingnodes = modeler.script.stream().findAll(CacheFilter(), False)
Managing your IBM Cloud account
Managing your IBM Cloud account You can manage your IBM Cloud account to view billing and usage, manage account users, and manage services. Required permissions : You must be the IBM Cloud account owner or administrator. To manage your IBM Cloud account, choose Administration > Account and billing > Account > Manage in IBM Cloud from IBM watsonx. Then from the IBM Cloud console, choose an option from the Manage menu. * Account: See [Adding orgs and spaces](https://cloud.ibm.com/docs/account?topic=account-orgsspacesusersorgsspacesusers) and [Managing resource groups](https://cloud.ibm.com/docs/account?topic=account-rgs). * Billing and Usage: See [How you're charged](https://cloud.ibm.com/docs/billing-usage?topic=billing-usage-chargescharges). * Access (IAM): See [Inviting users](https://cloud.ibm.com/docs/account?topic=account-access-getstarted). Learn more * [Activity Tracker events](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/at-events.html) * [Manage your settings](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/personal-settings.html) * [Set up IBM watsonx for your organization](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/setup-platform.html) * [Manage users and access](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/setup-access.html) * [IBM Cloud SAML Federation Guide](https://www.ibm.com/cloud/blog/ibm-cloud-saml-federation-guide) * [Delete your IBM Cloud account](https://dataplatform.cloud.ibm.com/docs/content/wsj/console/wdp_admin_console.htmldeletecloud) * [Check the status of IBM Cloud services](https://dataplatform.cloud.ibm.com/docs/content/wsj/troubleshoot/service-status.html) * [Configure private service endpoints](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/endpoints-vrf.html) Parent topic:[Administration](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/administer-accounts.html)
{{ document.title.text }}
Confidential data disclosure ![icon for intellectual property risk](https://dataplatform.cloud.ibm.com/docs/content/wsj/ai-risk-atlas/images/risk-intellectual-property.svg)Risks associated with inputTraining and tuning phaseIntellectual propertyTraditional Description Models might be trained or fine-tuned using confidential data or the company’s intellectual property, which could result in unwanted disclosure of that information. Why is confidential data disclosure a concern for foundation models? If not developed in accordance with data protection rules and regulations, the model might expose confidential information or IP in the generated output or through an adversarial attack. Parent topic:[AI risk atlas](https://dataplatform.cloud.ibm.com/docs/content/wsj/ai-risk-atlas/ai-risk-atlas.html)
The Concepts tab (SPSS Modeler)
The Concepts tab In the Text Analytics Workbench, you can use the Concepts tab to create and explore concepts as well as explore and tweak the extraction results. Concepts are the most basic level of extraction results available to use as building blocks, called descriptors, for your categories. Categories are a group of closely related ideas and patterns to which documents and records are assigned through a scoring process. Text mining is an iterative process in which extraction results are reviewed according to the context of the text data, fine-tuned to produce new results, and then reevaluated. Extraction results can be refined by modifying the linguistic resources. To simplify the process of fine-tuning your linguistic resources, you can perform common dictionary tasks directly from the Concepts tab. You can fine-tune other linguistic resources directly from the Resource editor tab. Figure 1. Concepts tab ![Concepts tab](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/tmwb_conceptview.png)
Using an Analysis node (SPSS Modeler)
Using an Analysis node You can assess the accuracy of the model using an Analysis node. From the Palette, under Outputs, place an Analysis node on the canvas and attach it to the C5.0 model nugget. Then right-click the Analysis node and select Run. Figure 1. Analysis node ![Analysis node](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/tut_drug_analysis.png)The Analysis node output shows that with this artificial dataset, the model correctly predicted the choice of drug for every record in the dataset. With a real dataset you are unlikely to see 100% accuracy, but you can use the Analysis node to help determine whether the model is acceptably accurate for your particular application. Figure 2. Analysis node output ![Analysis node output](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/tut_drug_analysis_output.png)
Values and data types (SPSS Modeler)
Values and data types CLEM expressions are similar to formulas constructed from values, field names, operators, and functions. The simplest valid CLEM expression is a value or a field name. Examples of valid values are: 3 1.79 'banana' Examples of field names are: Product_ID '$P-NextField' where Product is the name of a field from a market basket data set, '$P-NextField' is the name of a parameter, and the value of the expression is the value of the named field. Typically, field names start with a letter and may also contain digits and underscores (_). You can use names that don't follow these rules if you place the name within quotation marks. CLEM values can be any of the following: * Strings (for example, "c1", "Type 2", "a piece of free text") * Integers (for example, 12, 0, –189) * Real numbers (for example, 12.34, 0.0, –0.0045) * Date/time fields (for example, 05/12/2002, 12/05/2002, 12/05/02) It's also possible to use the following elements: * Character codes (for example, a or 3) * Lists of items (for example, [1 2 3], ['Type 1' 'Type 2']) Character codes and lists don't usually occur as field values. Typically, they're used as arguments of CLEM functions.
OPL models
OPL models You can build OPL models in the Decision Optimization experiment UI in watsonx.ai. In this section: * [Inputs and Outputs](https://dataplatform.cloud.ibm.com/docs/content/DO/DODS_Introduction/OPLmodels.html?context=cdpaas&locale=entopic_oplmodels__section_oplIO) * [Engine settings](https://dataplatform.cloud.ibm.com/docs/content/DO/DODS_Introduction/OPLmodels.html?context=cdpaas&locale=entopic_oplmodels__engsettings) To create an OPL model in the experiment UI, select in the model selection window. You can also import OPL models from a file or import a scenario .zip file that contains the OPL model and the data. If you import from a file or scenario .zip file, the data must be in .csv format. However, you can import other file formats that you have as project assets into the experiment UI. You can also import data sets including connected data into your project from the model builder in the [Prepare data view](https://dataplatform.cloud.ibm.com/docs/content/DO/DODS_Introduction/modelbuilderUI.htmlModelBuilderInterface__section_preparedata). For more information about the OPL language and engine parameters, see: * [OPL language reference manual](https://www.ibm.com/docs/en/SSSA5P_22.1.1/ilog.odms.ide.help/OPL_Studio/opllangref/topics/opl_langref_modeling_language.html) * [OPL Keywords](https://www.ibm.com/docs/en/SSSA5P_22.1.1/ilog.odms.ide.help/OPL_Studio/opllang_quickref/topics/opl_keywords_top.html) * [A list of CPLEX parameters](https://www.ibm.com/docs/en/SSSA5P_22.1.1/ilog.odms.cplex.help/CPLEX/Parameters/topics/introListTopical.html) * [A list of CPO parameters](https://www.ibm.com/docs/en/SSSA5P_22.1.1/ilog.odms.cpo.help/CP_Optimizer/Parameters/topics/paramcpoptimizer.html)
Forecasting bandwidth utilization (SPSS Modeler)
Forecasting bandwidth utilization An analyst for a national broadband provider is required to produce forecasts of user subscriptions to predict utilization of bandwidth. Forecasts are needed for each of the local markets that make up the national subscriber base. You'll use time series modeling to produce forecasts for the next three months for a number of local markets.
Creating nodes
Creating nodes Flows provide a number of ways to create nodes. These methods are summarized in the following table. Methods for creating nodes Table 1. Methods for creating nodes Method Return type Description s.create(nodeType, name) Node Creates a node of the specified type and adds it to the specified flow. s.createAt(nodeType, name, x, y) Node Creates a node of the specified type and adds it to the specified flow at the specified location. If either x < 0 or y < 0, the location is not set. s.createModelApplier(modelOutput, name) Node Creates a model applier node that's derived from the supplied model output object. For example, you can use the following script to create a new Type node in a flow: stream = modeler.script.stream() Create a new Type node node = stream.create("type", "My Type")
Functions handling blanks and null values (SPSS Modeler)
Functions handling blanks and null values Using CLEM, you can specify that certain values in a field are to be regarded as "blanks," or missing values. The following functions work with blanks. CLEM blank and null value functions Table 1. CLEM blank and null value functions Function Result Description @BLANK(FIELD) Boolean Returns true for all records whose values are blank according to the blank-handling rules set in an upstream Type node or Import node (Types tab). @LAST_NON_BLANK(FIELD) Any Returns the last value for FIELD that was not blank, as defined in an upstream Import or Type node. If there are no nonblank values for FIELD in the records read so far, $null$ is returned. Note that blank values, also called user-missing values, can be defined separately for each field. @NULL(FIELD) Boolean Returns true if the value of FIELD is the system-missing $null$. Returns false for all other values, including user-defined blanks. If you want to check for both, use @BLANK(FIELD) and@NULL(FIELD). undef Any Used generally in CLEM to enter a $null$ value—for example, to fill blank values with nulls in the Filler node. Blank fields may be "filled in" with the Filler node. In both Filler and Derive nodes (multiple mode only), the special CLEM function @FIELD refers to the current field(s) being examined.
rfmanalysisnode properties
rfmanalysisnode properties ![RFM Analysis node icon](https://dataplatform.cloud.ibm.com/docs/content/wsd/nodes/scripting_guide/clementine/images/rfm_analysis_icon.png)The Recency, Frequency, Monetary (RFM) Analysis node enables you to determine quantitatively which customers are likely to be the best ones by examining how recently they last purchased from you (recency), how often they purchased (frequency), and how much they spent over all transactions (monetary). rfmanalysisnode properties Table 1. rfmanalysisnode properties rfmanalysisnode properties Data type Property description recency field Specify the recency field. This may be a date, timestamp, or simple number. frequency field Specify the frequency field. monetary field Specify the monetary field. recency_bins integer Specify the number of recency bins to be generated. recency_weight number Specify the weighting to be applied to recency data. The default is 100. frequency_bins integer Specify the number of frequency bins to be generated. frequency_weight number Specify the weighting to be applied to frequency data. The default is 10. monetary_bins integer Specify the number of monetary bins to be generated. monetary_weight number Specify the weighting to be applied to monetary data. The default is 1. tied_values_method NextCurrent Specify which bin tied value data is to be put in. recalculate_bins AlwaysIfNecessary add_outliers flag Available only if recalculate_bins is set to IfNecessary. If set, records that lie below the lower bin will be added to the lower bin, and records above the highest bin will be added to the highest bin. binned_field RecencyFrequencyMonetary recency_thresholds value value Available only if recalculate_bins is set to Always. Specify the upper and lower thresholds for the recency bins. The upper threshold of one bin is used as the lower threshold of the next—for example, [10 30 60] would define two bins, the first bin with upper and lower thresholds of 10 and 30, with the second bin thresholds of 30 and 60. frequency_thresholds value value Available only if recalculate_bins is set to Always. monetary_thresholds value value Available only if recalculate_bins is set to Always.
Reading in source text (SPSS Modeler)
Reading in source text You can use the Language Identifier node to identify the natural language of a text field within your source data. The output of this node is a derived field that contains the detected language code. ![Language Identifier node](https://dataplatform.cloud.ibm.com/docs/content/wsd/images/ta_languageidentifier.png) Data for text mining can be in any of the standard formats that are used by SPSS Modeler flows, including databases or other "rectangular" formats that represent data in rows and columns. * To read in text from any of the standard data formats used by SPSS Modeler flows, such as a database with one or more text fields for customer comments, you can use an Import node. * When you're processing large amounts of data, which might include text in several different languages, use the Language Identifier node to identify the language used in a specific field.