itsjhuang/watsonx-docs-document-type huggingface dataset:
Traversing through nodes in a flow
Traversing through nodes in a flow A common requirement is to identify nodes that are either upstream or downstream of a particular node. The flow provides a number of methods that can be used to identify these nodes. These methods are summarized in the following table. Methods to identify upstream and downstream nodes Table 1. Methods to identify upstream and downstream nodes Method Return type Description s.iterator() Iterator Returns an iterator over the node objects that are contained in the specified flow. If the flow is modified between calls of the next() function, the behavior of the iterator is undefined. s.predecessorAt(node, index) Node Returns the specified immediate predecessor of the supplied node or None if the index is out of bounds. s.predecessorCount(node) int Returns the number of immediate predecessors of the supplied node. s.predecessors(node) List Returns the immediate predecessors of the supplied node. s.successorAt(node, index) Node Returns the specified immediate successor of the supplied node or None if the index is out of bounds. s.successorCount(node) int Returns the number of immediate successors of the supplied node. s.successors(node) List Returns the immediate successors of the supplied node.
Batch deployment input details for Spark models
Batch deployment input details for Spark models Follow these rules when you are specifying input details for batch deployments of Spark models. Data type summary table: Data Description Type Inline File formats N/A Parent topic:[Batch deployment input details by framework](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/deploy-batch-input-by-framework.html)
Examples
Examples The print keyword prints the arguments immediately following it. If the statement is followed by a comma, a new line isn't included in the output. For example: print "This demonstrates the use of a", print " comma at the end of a print statement." This will result in the following output: This demonstrates the use of a comma at the end of a print statement. The for statement iterates through a block of code. For example: mylist1 = ["one", "two", "three"] for lv in mylist1: print lv continue In this example, three strings are assigned to the list mylist1. The elements of the list are then printed, with one element of each line. This results in the following output: one two three In this example, the iterator lv takes the value of each element in the list mylist1 in turn as the for loop implements the code block for each element. An iterator can be any valid identifier of any length. The if statement is a conditional statement. It evaluates the condition and returns either true or false, depending on the result of the evaluation. For example: mylist1 = ["one", "two", "three"] for lv in mylist1: if lv == "two" print "The value of lv is ", lv else print "The value of lv is not two, but ", lv continue In this example, the value of the iterator lv is evaluated. If the value of lv is two, a different string is returned to the string that's returned if the value of lv is not two. This results in the following output: The value of lv is not two, but one The value of lv is two The value of lv is not two, but three
Inheritance
Inheritance The ability to inherit from classes is fundamental to object-oriented programming. Python supports both single and multiple inheritance. Single inheritance means that there can be only one superclass. Multiple inheritance means that there can be more than one superclass. Inheritance is implemented by subclassing other classes. Any number of Python classes can be superclasses. In the Jython implementation of Python, only one Java class can be directly or indirectly inherited from. It's not required for a superclass to be supplied. Any attribute or method in a superclass is also in any subclass and can be used by the class itself, or by any client as long as the attribute or method isn't hidden. Any instance of a subclass can be used wherever an instance of a superclass can be used; this is an example of polymorphism. These features enable reuse and ease of extension.
Integrating with AWS
Integrating with AWS You can configure an integration with the Amazon Web Services (AWS) platform to allow IBM watsonx users access to data sources from AWS. Before proceeding, make sure you have proper permissions. For example, you'll need to be able to create services and credentials in the AWS account. After you configure an integration, you'll see it under Service instances. You'll see a new AWS tab that lists your instances of Redshift and S3. To configure an integration with AWS: 1. Log on to the [AWS Console](https://aws.amazon.com/console/). 2. From the account drop-down at the upper right, select My Security Credentials. 3. Under Access keys (access key ID and secret access key), click Create New Access Key. 4. Copy the key ID and secret. Important: Write down your key ID and secret and store them in a safe place. 5. In IBM watsonx, under Administration > Cloud integrations, go to the AWS tab, enable integration, and then paste the access key ID and access key secret into the appropriate fields. 6. If you need to access Redshift, [configure firewall access](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/firewall-redshift.html). 7. Confirm that you can see your AWS services. From the main menu, choose Administration > Services > Services instances. Click the AWS tab to see those services. Now users who have credentials to your AWS services can [create connections](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/create-conn.html) to them by selecting them on the Add connection page. Then they can access data from those connections by [creating connected data assets](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/connected-data.html). Next steps * [Set up a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/projects.html) * [Create connections in a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/create-conn.html) Parent topic:[Integrations with other cloud platforms](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/int-cloud.html)
Data metadata (SPSS Modeler)
Data metadata This section describes how to set up the data model attributes based on pyspark.sql.StructField.
Regional availability for services and features
Regional availability for services and features IBM watsonx is deployed on the IBM Cloud multi-zone region network. The availability of services and features can vary across regional data centers. You can view the regional availability for every service in the [Services catalog](https://dataplatform.cloud.ibm.com/data/catalog?target=services&context=wx). Regional availability of the Watson Studio and Watson Machine Learning services Watsonx.ai includes the Watson Studio and Watson Machine Learning services to provide foundation and machine learning model tools. The Watson Studio and Watson Machine Learning services are available in the following regional data centers: * Dallas (us-south), in Texas US * Frankfurt (eu-de), in Germany Regional availability of foundation models The following table shows the IBM Cloud data centers where each foundation model is available. A checkmark indicates that the model is hosted in the region. Table 1. IBM Cloud data center support Model name Dallas Frankfurt flan-t5-xl-3b ✓ flan-t5-xxl-11b ✓ ✓ flan-ul2-20b ✓ ✓ gpt-neox-20b ✓ ✓ granite-13b-chat-v2 ✓ ✓ granite-13b-chat-v1 ✓ ✓ granite-13b-instruct-v2 ✓ ✓ granite-13b-instruct-v1 ✓ ✓ llama-2-13b-chat ✓ ✓ llama-2-70b-chat ✓ ✓ mpt-7b-instruct2 ✓ ✓ mt0-xxl-13b ✓ ✓ starcoder-15.5b ✓ ✓ Tool and environment limitations for the Frankfurt region Table 2. Frankfurt regional limitations Service Limitation Watson Studio If you need a Spark runtime, you must use the [Spark environment](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/jupyter-spark.html) in Watson Studio for the SPSS Modeler and notebook editor tools. Watson Studio [Synthetic Data Generator](https://dataplatform.cloud.ibm.com/docs/content/wsj/synthetic/synthetic_data_overview_sd.html) is not supported. Watson Studio [Tuning Studio](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-tuning-studio.html) is not supported. Regional availability of watsonx.governance Watsonx.governance Lite and Essentials plans are available only in the Dallas region. Regional availability of Watson OpenScale Watson OpenScale legacy plans are available only in the Frankfurt region. Regional availability of the Cloud Object Storage service The region for the Cloud Object Storage service is Global. Cloud Object Storage buckets for workspaces are Regional buckets. For more information, see [IBM Cloud docs: Cloud Object Storage endpoints and storage locations](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpoints). Learn more * [IBM Cloud docs: IBM Cloud global data centers](https://www.ibm.com/cloud/data-centers) * [Services in the IBM watsonx catalog](https://dataplatform.cloud.ibm.com/docs/content/svc-welcome/cloud-services.html) Parent topic:[Services and integrations](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/svc-int.html)
Batch deployment input details for Pytorch models
Batch deployment input details for Pytorch models Follow these rules when you are specifying input details for batch deployments of Pytorch models. Data type summary table: Data Description Type inline, data references File formats .zip archive that contains JSON files Data sources Input or output data references: * Local or managed assets from the space * Connected (remote) assets: Cloud Object Storage If you are specifying input/output data references programmatically: * Data source reference type depends on the asset type. Refer to the Data source reference types section in [Adding data assets to a deployment space](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-space-add-assets.html). * If you deploy Pytorch models with ONNX format, specify the keep_initializers_as_inputs=True flag and set opset_version to 9 (always set opset_version to the most recent version that is supported by the deployment runtime). torch.onnx.export(net, x, 'lin_reg1.onnx', verbose=True, keep_initializers_as_inputs=True, opset_version=9) Note: The environment variables parameter of deployment jobs is not applicable. Parent topic:[Batch deployment input details by framework](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/deploy-batch-input-by-framework.html)
Linear-AS node (SPSS Modeler)
Linear-AS node Linear regression is a common statistical technique for classifying records based on the values of numeric input fields. Linear regression fits a straight line or surface that minimizes the discrepancies between predicted and actual output values. Requirements. Only numeric fields and categorical predictors can be used in a linear regression model. You must have exactly one target field (with the role set to Target) and one or more predictors (with the role set to Input). Fields with a role of Both or None are ignored, as are non-numeric fields. (If necessary, non-numeric fields can be recoded using a Derive node.) Strengths. Linear regression models are relatively simple and give an easily interpreted mathematical formula for generating predictions. Because linear regression is a long-established statistical procedure, the properties of these models are well understood. Linear models are also typically very fast to train. The Linear node provides methods for automatic field selection in order to eliminate non-significant input fields from the equation. Note: In cases where the target field is categorical rather than a continuous range, such as yes/no or churn/don't churn, logistic regression can be used as an alternative. Logistic regression also provides support for non-numeric inputs, removing the need to recode these fields.
Global functions (SPSS Modeler)
Global functions The functions @MEAN, @SUM, @MIN, @MAX, and @SDEV work on, at most, all of the records read up to and including the current one. In some cases, however, it is useful to be able to work out how values in the current record compare with values seen in the entire data set. Using a Set Globals node to generate values across the entire data set, you can access these values in a CLEM expression using the global functions. For example, @GLOBAL_MAX(Age) returns the highest value of Age in the data set, while the expression (Value - @GLOBAL_MEAN(Value)) / @GLOBAL_SDEV(Value) expresses the difference between this record's Value and the global mean as a number of standard deviations. You can use global values only after they have been calculated by a Set Globals node. CLEM global functions Table 1. CLEM global functions Function Result Description @GLOBAL_MAX(FIELD) Number Returns the maximum value for FIELD over the whole data set, as previously generated by a Set Globals node. FIELD must be the name of a numeric, date/time/datetime, or string field. If the corresponding global value has not been set, an error occurs. @GLOBAL_MIN(FIELD) Number Returns the minimum value for FIELD over the whole data set, as previously generated by a Set Globals node. FIELD must be the name of a numeric, date/time/datetime, or string field. If the corresponding global value has not been set, an error occurs. @GLOBAL_SDEV(FIELD) Number Returns the standard deviation of values for FIELD over the whole data set, as previously generated by a Set Globals node. FIELD must be the name of a numeric field. If the corresponding global value has not been set, an error occurs. @GLOBAL_MEAN(FIELD) Number Returns the mean average of values for FIELD over the whole data set, as previously generated by a Set Globals node. FIELD must be the name of a numeric field. If the corresponding global value has not been set, an error occurs. @GLOBAL_SUM(FIELD) Number Returns the sum of values for FIELD over the whole data set, as previously generated by a Set Globals node. FIELD must be the name of a numeric field. If the corresponding global value has not been set, an error occurs.
User Input node (SPSS Modeler)
User Input node The User Input node provides an easy way for you to create synthetic data--either from scratch or by altering existing data. This is useful, for example, when you want to create a test dataset for modeling.
Creating a Federated Learning experiment
Creating a Federated Learning experiment Learn how to create a Federated Learning experiment to train a machine learning model. Watch this short overview video of how to create a Federated Learning experiment. Video disclaimer: Some minor steps and graphical elements in this video might differ from your platform. This video provides a visual method to learn the concepts and tasks in this documentation. Follow these steps to create a Federated Learning experiment: * [Set up your system](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fl-setup.html) * [Creating the initial model](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fl-models.html) * [Create the data handler](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fl-handler.html) * [Starting the aggregator (Admin)](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fl-agg.html) * [Connecting to the aggregator (Party)](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fl-conn.html) * [Monitoring and saving the model](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fl-mon.html) Parent topic:[IBM Federated Learning](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fed-lea.html)
Extension Output node (SPSS Modeler)
Extension Output node With the Extension Output node, you can run R scripts or Python for Spark scripts to produce output. After adding the node to your canvas, double-click the node to open its properties.
Creating SPSS Modeler jobs
Creating SPSS Modeler jobs You can create a job to run an SPSS Modeler flow. To create an SPSS Modeler job: 1. In SPSS Modeler, click the Create a job icon ![the jobs icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/images/Run-schedule_Blue.png) from the toolbar and select Create a job. A wizard will appear. Click Next to proceed through each page of the wizard as described here. 2. Define the job details by entering a name and a description (optional). If desired, you can also specify retention settings for the job. Select Job run retention settings to set how long to retain finished job runs and job run artifacts such as logs. You can select one of the following retention methods. Be mindful when changing the default as too many job run files can quickly use up project storage. * By duration (days). Specify the number of days to retain job runs and job artifacts. The retention value is set to 7 days by default (the last 7 days of job runs retained). * By amount. Specify the last number of finished job runs and job artifacts to keep. The retention value is set to 200 jobs by default. 3. On the Flow parameters page, you can set values for flow parameters if any exist for the flow. They are, in effect, user-defined variables that are saved and persisted with the flow. Parameters are often used in scripting to control the behavior of the script by providing information about fields and values that don't need to be hard coded in the script. See [Setting properties for flows](https://dataplatform.cloud.ibm.com/docs/content/wsd/flow_properties.html) for more information. For example, your flow might contain a parameter called age_param that you choose to set to 40 here, and a parameter called bp_param you might set to HIGH. 4. On the Configuration page, you can choose whether the job will run the entire flow or one or more branches of the flow. 5. On the Schedule page, you can optionally add a one-time or repeating schedule. If you define a start day and time without selecting Repeat, the job will run exactly one time at the specified day and time. If you define a start date and time and you select Repeat, the job will run for the first time at the timestamp indicated in the Repeat section. You can't change the time zone; the schedule uses your web browser's time zone setting. If you exclude certain weekdays, the job might not run as you would expect. The reason might be due to a discrepancy between the time zone of the user who creates the schedule, and the time zone of the compute node where the job runs. 6. Optionally turn on notifications for the job. You can select the type of alerts to receive. 7. Review the job settings. Click Save to create the job. The SPSS Modeler job is listed under Jobs in your project. Learn more * [Viewing job details](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/jobs.htmlview-job-details) * [SPSS Modeler documentation](https://dataplatform.cloud.ibm.com/docs/content/wsd/spss-modeler.html) Parent topic: [Jobs](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/jobs.html)
t-SNE charts
t-SNE charts T-distributed Stochastic Neighbor Embedding (t-SNE) is a machine learning algorithm for visualization. t-SNE charts model each high-dimensional object by a two-or-three dimensional point in such a way that similar objects are modeled by nearby points and dissimilar objects are modeled by distant points with high probability.
One-Class SVM node (SPSS Modeler)
One-Class SVM node The One-Class SVM© node uses an unsupervised learning algorithm. The node can be used for novelty detection. It will detect the soft boundary of a given set of samples, to then classify new points as belonging to that set or not. This One-Class SVM modeling node is implemented in Python and requires the scikit-learn© Python library. For details about the scikit-learn library, see [Support Vector Machines](http://scikit-learn.org/stable/modules/svm.htmlsvm-outlier-detection)^1^. The Modeling tab on the palette contains the One-Class SVM node and other Python nodes. Note: One-Class SVM is used for usupervised outlier and novelty detection. In most cases, we recommend using a known, "normal" dataset to build the model so the algorithm can set a correct boundary for the given samples. Parameters for the model – such as nu, gamma, and kernel – impact the result significantly. So you may need to experiment with these options until you find the optimal settings for your situation. ^1^Smola, Schölkopf. "A Tutorial on Support Vector Regression." Statistics and Computing Archive, vol. 14, no. 3, August 2004, pp. 199-222. (http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.114.4288)
{{ document.title.text }}
Data privacy rights ![icon for privacy risk](https://dataplatform.cloud.ibm.com/docs/content/wsj/ai-risk-atlas/images/risk-privacy.svg)Risks associated with inputTraining and tuning phasePrivacyAmplified Description In some countries, privacy laws give individuals the right to access, correct, verify, or remove certain types of information that companies hold or process about them. Tracking the usage of an individual’s personal information in training a model and providing appropriate rights to comply with such laws can be a complex endeavor. Why is data privacy rights a concern for foundation models? The identification or improper usage of data could lead to violation of privacy laws. Improper usage or a request for data removal could force organizations to retrain the model, which is expensive. In addition, business entities could face fines, reputational harms, and other legal consequences if they fail to comply with data privacy rules and regulations. Example Right to Be Forgotten (RTBF) As stated in the article, laws in multiple locales, including Europe (GDPR); Canada (CPPA); and Japan (APPI), grant users’ rights for their personal data to be “forgotten” by technology (Right To Be Forgotten). However, the emerging and increasingly popular AI (LLMs) services present new challenges for the right to be forgotten (RTBF). According to Data61’s research, the only way for users to identify usage of their personal information in an LLM is “by either inspecting the original training dataset or perhaps prompting the model.” However, training data is either not public or companies do not disclose it, citing safety and other concerns, and guardrails may prevent users from accessing the information via prompting. Due to these barriers, users cannot initiate RTBF procedures and companies deploying LLMs may be unable to meet RTBF laws. Sources: [Zhang et al., Sept 2023](https://arxiv.org/pdf/2307.03941.pdf) Example Lawsuit About LLM Unlearning According to the report, a lawsuit was filed against Google that alleges the use of copyright material and personal information as training data for its AI systems, which includes its Bard chatbot. Opt-out and deletion rights are guaranteed rights for California residents under the CCPA and children in the United States below 13 under the COPPA. The plaintiffs allege that because there is no way for Bard to “unlearn” or fully remove all the scraped PI it has been fed. The plaintiffs note that Bard’s privacy notice states that Bard conversations cannot be deleted by the user once they have been reviewed and annotated by the company and may be kept up to 3 years, which plaintiffs allege further contributes to non-compliance with these laws. Sources: [Reuters, July 2023](https://www.reuters.com/legal/litigation/google-hit-with-class-action-lawsuit-over-ai-data-scraping-2023-07-11/) [J.L. v. Alphabet Inc., July 2023](https://fingfx.thomsonreuters.com/gfx/legaldocs/myvmodloqvr/GOOGLE%20AI%20LAWSUIT%20complaint.pdf) Parent topic:[AI risk atlas](https://dataplatform.cloud.ibm.com/docs/content/wsj/ai-risk-atlas/ai-risk-atlas.html)