ibm-research/WatsonxDocsQARetrieval huggingface dataset:
# SAP ASE connection #

To access your data in SAP ASE, create a connection asset for it\.

SAP ASE is a relational model database server\. SAP ASE was formerly Sybase\.

## Supported versions ##

SAP Sybase ASE 11\.5\+, 16\.0\+

## Create a connection to SAP ASE ##

To create the connection asset, you need these connection details:

<!-- <ul> -->

 *  Database name
 *  Hostname or IP address
 *  Port number
 *  Username and password
 *  SSL certificate (if required by the database server)

<!-- </ul> -->

For **Private connectivity**, to connect to a database that is not externalized to the internet (for example, behind a firewall), you must set up a [secure connection](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/securingconn.html)\.

### Choose the method for creating a connection based on where you are in the platform ###

**In a project** : Click **Assets > New asset > Connect to a data source**\. See [Adding a connection to a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/create-conn.html)\.

**In a deployment space** : Click **Add to space > Connection**\. See [Adding connections to a deployment space](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-space-add-assets.html)\.

**In the Platform assets catalog** : Click **New connection**\. See [Adding platform connections](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/platform-conn.html)\.

### Next step: Add data assets from the connection ###

<!-- <ul> -->

 *  See [Add data from a connection in a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/connected-data.html)\.

<!-- </ul> -->

## Where you can use this connection ##

You can use SAP ASE connections in the following workspaces and tools:

**Projects**

<!-- <ul> -->

 *  SPSS Modeler
 *  Synthetic Data Generator

<!-- </ul> -->

**Catalogs**

<!-- <ul> -->

 *  Platform assets catalog

<!-- </ul> -->

## SAP ASE setup ##

[Get Started with SAP ASE](https://www.sap.com/canada/products/sybase-ase/get-started.html)

### Running SQL statements ###

To ensure that your SQL statements run correctly, refer to the [SAP ASE documentation](https://help.sap.com/viewer/product/SAP_ASE/16.0.4.1/en-US?task=whats_new_task) for the correct syntax\.

## Learn more ##

[SAP ASE technical information](https://www.sap.com/canada/products/sybase-ase/technical-information.html)

**Parent topic:**[Supported connections](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/conn_types.html)

<!-- </article "role="article" "> -->
# SAP IQ connection #

To access your data in SAP IQ, create a connection asset for it\.

SAP IQ is a column\-based, petabyte scale, relational database software system used for business intelligence, data warehousing, and data marts\. SAP IQ was formerly Sybase IQ\.

## Create a connection to SAP IQ ##

To create the connection asset, you need these connection details:

<!-- <ul> -->

 *  Database name
 *  Hostname or IP address
 *  Port number
 *  Username and password
 *  SSL certificate (if required by the database server)

<!-- </ul> -->

For **Private connectivity**, to connect to a database that is not externalized to the internet (for example, behind a firewall), you must set up a [secure connection](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/securingconn.html)\.

### Choose the method for creating a connection based on where you are in the platform ###

**In a project** : Click **Assets > New asset > Connect to a data source**\. See [Adding a connection to a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/create-conn.html)\.

**In a deployment space** : Click **Add to space > Connection**\. See [Adding connections to a deployment space](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-space-add-assets.html)\.

**In the Platform assets catalog** : Click **New connection**\. See [Adding platform connections](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/platform-conn.html)\.

### Next step: Add data assets from the connection ###

<!-- <ul> -->

 *  See [Add data from a connection in a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/connected-data.html)\.

<!-- </ul> -->

## Where you can use this connection ##

You can use SAP IQ connections in the following workspaces and tools: **Projects**

<!-- <ul> -->

 *  SPSS Modeler
 *  Synthetic Data Generator

<!-- </ul> -->

**Catalogs**

<!-- <ul> -->

 *  Platform assets catalog

<!-- </ul> -->

## SAP IQ setup ##

[Get Started with SAP IQ](https://www.sap.com/canada/products/sybase-iq-big-data-management/get-started.html)

## Restriction ##

You can use this connection only for source data\. You cannot write to data or export data with this connection\.

### Running SQL statements ###

To ensure that your SQL statements run correctly, refer to the [SAP IQ SQL Reference](https://help.sap.com/docs/SAP_IQ/a898e08b84f21015969fa437e89860c8/7b5bd4e8cdcb4593aba6f2895572b0a9.html) for the correct syntax\.

## Learn more ##

[SAP IQ technical information](https://www.sap.com/canada/products/sybase-iq-big-data-management/technical-information.html)

**Parent topic:**[Supported connections](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/conn_types.html)

<!-- </article "role="article" "> -->
# SAP OData connection #

To access your data in SAP OData, create a connection asset for it\.

Use the SAP OData connection to extract data from a SAP system through its exposed OData services\.

## Supported SAP OData products ##

The SAP OData connection is supported on SAP products that support the OData protocol version 2\. Example products are S4/HANA (on premises or cloud), ERP, and CRM\.

## Create a connection to SAP OData ##

To create the connection asset, you need these connection details:

Credentials type:

<!-- <ul> -->

 *  API Key
 *  Basic
 *  None

<!-- </ul> -->

Encryption:  
SSL certificate (if required by the database server)

For **Private connectivity**, to connect to a database that is not externalized to the internet (for example, behind a firewall), you must set up a [secure connection](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/securingconn.html)\.

### Choose the method for creating a connection based on where you are in the platform ###

**In a project** : Click **Assets > New asset > Connect to a data source**\. See [Adding a connection to a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/create-conn.html)\.

**In a deployment space** : Click **Add to space > Connection**\. See [Adding connections to a deployment space](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-space-add-assets.html)\.

**In the Platform assets catalog** : Click **New connection**\. See [Adding platform connections](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/platform-conn.html)\.

### Next step: Add data assets from the connection ###

<!-- <ul> -->

 *  See [Add data from a connection in a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/connected-data.html)\.

<!-- </ul> -->

## Where you can use this connection ##

You can use the SAP OData connection in the following workspaces and tools:

**Projects**

<!-- <ul> -->

 *  Data Refinery
 *  SPSS Modeler
 *  Synthetic Data Generator

<!-- </ul> -->

**Catalogs**

<!-- <ul> -->

 *  Platform assets catalog

<!-- </ul> -->

## SAP OData setup ##

See [Prerequisites for using the SAP ODATA Connector](https://www.ibm.com/support/pages/node/886655) for the SAP Gateway setup instructions\.

## Restrictions ##

<!-- <ul> -->

 *  For Data Refinery, you can use this connection only as a source\. You cannot use this connection as a target connection or as a target connected data asset\.
 *  For SPSS Modeler, you cannot create new entity sets\.

<!-- </ul> -->

**Parent topic:**[Supported connections](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/conn_types.html)

<!-- </article "role="article" "> -->
# SingleStoreDB connection #

To access your data in SingleStoreDB, create a connection asset for it\.

SingleStoreDB is a fast, distributed, and highly scalable cloud\-based SQL database\. You can use SingleStoreDB to power real\-time and data\-intensive applications\.

Use SingleStoreDB and watsonx\.ai for generative AI applications\. Benefits include semantic search, fast ingest, and low\-latency response times for foundation models and traditional machine learning\.

## Create a connection to SingleStoreDB ##

To create the connection asset, you need these connection details:

<!-- <ul> -->

 *  Database name
 *  Hostname or IP address
 *  Port number
 *  Username and password
 *  SSL certificate (if required by the database server)

<!-- </ul> -->

### Choose the method for creating a connection based on where you are in the platform ###

**In a project**  
Click **Assets > New asset > Connect to a data source**\. See [Adding a connection to a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/create-conn.html)\.

**In a deployment space**  
Click **Add to space > Connection**\. See [Adding connections to a deployment space](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-space-add-assets.html)\.

**In the Platform assets catalog** : Click **New connection**\. See [Adding platform connections](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/platform-conn.html)\.

### Next step: Add data assets from the connection ###

<!-- <ul> -->

 *  See [Add data from a connection in a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/connected-data.html)\.

<!-- </ul> -->

## Where you can use this connection ##

You can use the SingleStoreDB connection in the following workspaces and tools:

**Projects**

<!-- <ul> -->

 *  Data Refinery
 *  Decision Optimization
 *  SPSS Modeler

<!-- </ul> -->

**Catalogs**

<!-- <ul> -->

 *  Platform assets catalog

<!-- </ul> -->

## SingleStoreDB setup ##

To set up SingleStoreDB, see [Getting Started with SingleStoreDB Cloud](https://docs.singlestore.com/cloud/getting-started-with-singlestoredb-cloud/)\.

### Running SQL statements ###

To ensure that your SQL statements run correctly, refer to the SingleStore Docs [SQL Reference](https://docs.singlestore.com/db/v8.1/reference/sql-reference/) for the correct syntax\.

## Learn more ##

<!-- <ul> -->

 *  [SingleStoreDB Cloud](https://docs.singlestore.com/)
 *  [SingleStoreDB with IBM](https://www.ibm.com/products/singlestore) for information about the IBM partnership with SingleStoreDB that provides a single source of procurement, support, and security\.

<!-- </ul> -->

**Parent topic**: [Supported connections](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/conn_types.html)

<!-- </article "role="article" "> -->
# Snowflake connection #

To access your data in Snowflake, you must create a connection asset for it\.

Snowflake is a cloud\-based data storage and analytics service\.

## Create a connection to Snowflake ##

To create the connection asset, you need the following connection details:

<!-- <ul> -->

 *  Account name: The full name of your account
 *  Database name
 *  Role: The default access control role to use in the Snowflake session
 *  Warehouse: The virtual warehouse

<!-- </ul> -->

### Credentials ###

Authentication method:

<!-- <ul> -->

 *  Username and password
 *  Key\-Pair: Enter the contents of the private key and the key passphrase (if configured)\. These properties must be set up by the Snowflake administrator\. For information, see [Key Pair Authentication & Key Pair Rotation](https://docs.snowflake.com/en/user-guide/key-pair-auth) in the Snowflake documentation\.
 *  Okta URL endpoint: If your company uses native Okta SSO authentication, enter the Okta URL endpoint for your Okta account\. Example: `https://<okta_account_name>.okta.com`\. Leave this field blank if you want to use the default authentication of Snowflake\. For information about federated authentication provided by Okta, see [Native SSO](https://docs.snowflake.com/en/user-guide/admin-security-fed-auth-use.html#native-sso-okta-only)\.

<!-- </ul> -->

### Choose the method for creating a connection based on where you are in the platform ###

**In a project** : Click **Assets > New asset > Connect to a data source**\. See [Adding a connection to a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/create-conn.html)\.

**In a deployment space** : Click **Add to space > Connection**\. See [Adding connections to a deployment space](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-space-add-assets.html)\.

**In the Platform assets catalog** : Click **New connection**\. See [Adding platform connections](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/platform-conn.html)\.

### Next step: Add data assets from the connection ###

<!-- <ul> -->

 *  See [Add data from a connection in a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/connected-data.html)\.

<!-- </ul> -->

## Where you can use this connection ##

You can use Snowflake connections in the following workspaces and tools:

**Projects**

<!-- <ul> -->

 *  Data Refinery
 *  Decision Optimization
 *  SPSS Modeler
 *  Synthetic Data Generator

<!-- </ul> -->

**Catalogs**

<!-- <ul> -->

 *  Platform assets catalog

<!-- </ul> -->

## Snowflake setup ##

[General Configuration ](https://docs.snowflake.com/en/user-guide/gen-conn-config.html)

### Running SQL statements ###

To ensure that your SQL statements run correctly, refer to the [Snowflake SQL Command Reference](https://docs.snowflake.com/en/sql-reference-commands.html) for the correct syntax\.

## Learn more ##

[Snowflake in 20 Minutes](https://docs.snowflake.com/en/user-guide/getting-started-tutorial.html)

**Parent topic:**[Supported connections](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/conn_types.html)

<!-- </article "role="article" "> -->
# Tableau connection #

To access your data in Tableau, you must create a connection asset for it\.

Tableau is an interactive data visualization platform\.

## Supported products ##

Tableau Server 2020\.3\.3 and Tableau Cloud

## Create a connection to Tableau ##

To create the connection asset, you need the following connection details:

<!-- <ul> -->

 *  Hostname or IP address
 *  Port number
 *  Site: The name of the Tableau site to use
 *  For **Authentication method**, you need either a username and password or an Access token (with Access token name and Access token secret)\.
 *  SSL certificate (if required by the database server)

<!-- </ul> -->

### Choose the method for creating a connection based on where you are in the platform ###

**In a project** : Click **Assets > New asset > Connect to a data source**\. See [Adding a connection to a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/create-conn.html)\.

**In a deployment space** : Click **Add to space > Connection**\. See [Adding connections to a deployment space](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-space-add-assets.html)\.

**In the Platform assets catalog** : Click **New connection**\. See [Adding platform connections](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/platform-conn.html)\.

### Next step: Add data assets from the connection ###

<!-- <ul> -->

 *  See [Add data from a connection in a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/connected-data.html)\.

<!-- </ul> -->

## Where you can use this connection ##

You can use Tableau connections in the following workspaces and tools: **Projects**

<!-- <ul> -->

 *  SPSS Modeler
 *  Synthetic Data Generator

<!-- </ul> -->

**Catalogs**

<!-- <ul> -->

 *  Platform assets catalog

<!-- </ul> -->

## Tableau setup ##

<!-- <ul> -->

 *  [Get Started with Tableau Server on Linux](https://help.tableau.com/current/server-linux/en-us/get_started_server.htm)
 *  [Get Started with Tableau Server on Windows](https://help.tableau.com/current/server/en-us/get_started_server.htm)
 *  [Get Started with Tableau Cloud](https://help.tableau.com/current/online/en-us/to_get_started.htm)

<!-- </ul> -->

## Restriction ##

You can use this connection only for source data\. You cannot write to data or export data with this connection\.

### Running SQL statements ###

To ensure that your SQL statements run correctly, refer to the [Run Initial SQL](https://help.tableau.com/current/online/en-us/connect_basic_initialsql.htm) for the correct syntax\.

## Learn more ##

<!-- <ul> -->

 *  [Tableau](https://www.tableau.com/)
 *  [SSL for Tableau Server on Linux](https://help.tableau.com/current/server-linux/en-us/ssl.htm)
 *  [SSL for Tableau Server on Windows](https://help.tableau.com/current/server/en-us/ssl.htm)
 *  [Security in Tableau Cloud](https://help.tableau.com/current/online/en-us/to_security.htm)

<!-- </ul> -->

**Parent topic:**[Supported connections](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/conn_types.html)

<!-- </article "role="article" "> -->
# Creating jobs in Data Refinery #

You can create a job to run a Data Refinery flow directly in Data Refinery\.

To create a Data Refinery flow job:

<!-- <ol> -->

1.  In Data Refinery, click the Jobs icon ![the jobs icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/images/Run-schedule_Blue.png) from the Data Refinery toolbar and select **Save and create a job**\.
2.  Define the job details by entering a name and a description (optional)\.
3.  On the Configure page, select an environment runtime for the job, and optionally modify the job retention settings\.
4.  On the Schedule page, you can optionally add a one\-time or repeating schedule\.
    
    If you define a start day and time without selecting **Repeat**, the job will run exactly one time at the specified day and time. If you define a start date and time and you select **Repeat**, the job will run for the first time at the timestamp indicated in the Repeat section.
    
    You can't change the time zone; the schedule uses your web browser's time zone setting. If you exclude certain weekdays, the job might not run as you would expect. The reason might be due to a discrepancy between the time zone of the user who creates the schedule, and the time zone of the compute node where the job runs.
5.  Optional: Set up notifications for the job\. You can select the type of alerts to receive\.
6.  Review the job settings\. Then, create the job and run it immediately, or create the job and run it later\.
    
    The Data Refinery flow job is listed in the **Jobs** in your project.

<!-- </ol> -->

## Learn more ##

<!-- <ul> -->

 *  [Compute resource options for Data Refinery in projects](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/spark-dr-envs.html)
 *  [Viewing job details](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/jobs.html#view-job-details)
 *  [Refining data](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/refining_data.html)

<!-- </ul> -->

**Parent topic**: [Jobs](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/jobs.html)

<!-- </article "role="article" "> -->
# Creating jobs for Pipelines #

You can create jobs for Pipelines\.

To create a Pipelines job:

<!-- <ol> -->

1.  Open your Pipelines asset from the project\.
2.  Click **Run pipeline > Create a job**\.
3.  On the **Create a job** page, you can choose the asset version that you'd like to run\. The most recently saved version of the Pipelines is used by default\.
4.  Give a name and optional description for your job\. Click **next**\.
5.  Define your IAM API key\. The most recently used API key is used by default\. If you'd like to use a new API key, click **Generate new API key**\. Click **next**\.
6.  You can schedule your job by toggling *Schedule off* to *Schedule to run*\. You can choose either or both options:  
    
    <!-- <ul> -->
    
     *  *Start on*: Choose a date for your scheduled job to run. The time zone is GMT-0400 (Eastern Daylight Time). If you do not choose a start date, the job will never run automatically and must be started manually.  
        
     *  *Repeat*: You can choose to schedule the repeated frequency (every minute to every month), exclude running the job on certain days, and choose an end date. If you do not choose to repeat the job, it runs one time if a start date is given, or does not run.
    
    <!-- </ul> -->
    
7.  Review your job settings and click **Create**\. The Pipelines job is listed under **Jobs** in your project\.

<!-- </ol> -->

## Learn more ##

<!-- <ul> -->

 *  [Viewing jobs across projects](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/job-views-projs.html)

<!-- </ul> -->

**Parent topic:**[Creating and managing jobs](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/jobs.html)

<!-- </article "role="article" "> -->
# Downloading data assets from a project #

You can download data assets from a project to your local system\.

Important: Take care when you download assets\. Collaborators can upload any type of file into a project, including files that have malware or other types of malicious code\.

**Required permissions** : You must have the **Editor** or **Admin** role in the project to download an asset\.

## Download a data asset ##

To download a data asset that is in the project's storage, select **Download** from the ACTION menu next to the asset name\.

For an alternate method of downloading data assets for a project, select **Files** in the Data side panel\. Checkmark the data asset and select **Download** from the ACTION menu in the side panel\.

## Download a connected data asset ##

To download a *connected data asset*, use Data Refinery to run a job that saves the file as the output of a Data Refinery flow\. The output of the Data Refinery flow is a new CSV file in the project’s storage\.

<!-- <ol> -->

1.  Click the asset name to open it\.
2.  Click **Prepare data**\.
3.  From the **Jobs** menu, click **Save and create a job**\. Enter a job name and click **Create and Run**\.
4.  Go back to the **Assets** page\. Refresh the page\. By default, the downloadable asset is named *table\-name*\_shaped\.csv\.
5.  Choose **Download** from the ACTION menu next to the asset name\.

<!-- </ol> -->

## Learn more ##

<!-- <ul> -->

 *  [Download a data asset from a catalog](https://dataplatform.cloud.ibm.com/docs/content/wsj/catalog/download.html)
 *  [Data Refinery](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/refining_data.html)

<!-- </ul> -->

**Parent topic:**[Projects](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/manage-projects.html)

<!-- </article "role="article" "> -->
# Adding a connected folder asset to a project #

You can create a connected folder asset based on a path within an IBM Cloud Object Storage system that is accessed through a connection\. You can view the files and subfolders that share the path with the connected folder asset\. The files that you can view within the connected folder asset are not themselves data assets\. For example, you can create a connected folder asset for a path that contains news feeds that are continuously updated\.

**Required permissions** : You must have the **Admin** or **Editor** role in the project to add a connected folder asset\.

Watch this video to see how to add a connected folder asset in a project, then follow the steps below the video\.

Video disclaimer: Some minor steps and graphical elements in this video might differ from your platform\.

This video provides a visual method to learn the concepts and tasks in this documentation\.

To add a connected folder asset from a connection to a project:

<!-- <ol> -->

1.  If necessary, [create a connection asset](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/create-conn.html)\. Include an Access Key and a Secret Key to your IBM Cloud Object Storage connection to enable the downloading of files within the connected folder asset\. If you're using an existing IBM Cloud Object Storage connection asset that doesn't have an Access Key and Secret Key, edit the connection asset and add them\.
2.  Click **Import assets > Connected data**\.
3.  Select an existing connection asset as the source of the data\.
4.  Select the folder you want and click **Import**\.
5.  Type a name and description\.
6.  Click **Create**\. The connected folder asset appears on the project **Assets** page in the **Data assets** category\.

<!-- </ol> -->

Click the connected folder asset name to view the contents of the connected folder asset\. Click the eye (![eye icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/console/images/visibility-on.svg)) icon next to a file name to view the contents of the files within the folder that have these formats:

<!-- <ul> -->

 *  CSV
 *  JSON
 *  Parquet

<!-- </ul> -->

You can refine the files *within* a connected folder asset and then save the result as a data asset\. While viewing the connected folder asset, select a file and then click **Prepare data**\.

You can view the files within the connected folder asset if the IBM Cloud Object Storage connection asset that's associated with the connected folder asset has an Access Key and a Secret Key (also known as HMAC credentials)\. For more information about HMAC credentials, see [IBM Cloud Object Storage Service credentials](https://console.bluemix.net/docs/services/cloud-object-storage/iam/service-credentials.html#service-credentials)\.

## Next steps ##

<!-- <ul> -->

 *  [Refining a file within the folder](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/refining_data.html)

<!-- </ul> -->

**Parent topic:**[Adding data to a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/add-data-project.html)

<!-- </article "role="article" "> -->
# Getting and preparing data in a project #

After you create a project, or join one, the next step is to add data to the project and prepare the data for analysis\.

**Required permissions** : You must have the **Admin** or **Editor** role in a project to add or prepare data\.

You can add data assets from your local system, from a catalog, from the Samples, or from connections to data sources\.

You can add these types of data assets to a project:

<!-- <ul> -->

 *  Data assets from files from your local system, including structured data, unstructured data, and images\. The files are stored in the project's IBM Cloud Object Storage bucket\.
 *  Connection assets that contain information for connecting to data sources\. You can add connections to IBM or third\-party data sources\. See [Supported connections](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/conn_types.html)\.
 *  Connected data assets that specify a table, view, or file that is accessed through a connection to a data source\.
 *  Connected folder assets that specify a path in IBM Cloud Object Storage\.

<!-- </ul> -->

To get started quickly, take a tutorial\. See [Quick start tutorials](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/quickstart-tutorials.html)\.

To [refine data](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/refining_data.html) by cleansing and shaping it, you can:

<!-- <ul> -->

 *  Select the **Prepare data** tile on your watsonx home page\.
 *  Add the data to the project, then open the data asset and click **Prepare data**\.

<!-- </ul> -->

To [manage feature groups](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/feature-group.html) for a data asset, open the data asset and go to its **Feature group** page\.

To create [synthetic data](https://dataplatform.cloud.ibm.com/docs/content/wsj/synthetic/synthetic_data_overview_sd.html), you can:

<!-- <ul> -->

 *  Select the **Prepare data** tile on your watsonx home page\.
 *  Select the **Generate synthetic tabular data** tile\.

<!-- </ul> -->

## Learn more ##

<!-- <ul> -->

 *  [Create a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/projects.html)
 *  [Adding data to a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/add-data-project.html)
 *  [Refining data with Data Refinery](https://dataplatform.cloud.ibm.com/docs/content/wsj/refinery/refining_data.html)
 *  [Adding connections to the Platform assets catalog](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/platform-conn.html)
 *  [Manage feature groups (beta)](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/feature-group.html)
 *  [Creating synthetic data](https://dataplatform.cloud.ibm.com/docs/content/wsj/synthetic/synthetic_data_overview_sd.html)

<!-- </ul> -->

<!-- </article "role="article" "> -->
# Importing a project #

You can create a project that is preloaded with assets by importing the project\.

## Requirements ##

**A local file of a previously exported project** : Importing a project from a local file is a method of copying a project\. You can import a project from a file on your local system only if the ZIP file that you select was exported from a IBM watsonx project as a compressed file\. You can import only projects that you exported from watsonx\.ai\. You cannot import a compressed file that was exported from a Cloud Pak for Data as a Service project\.

: If the exported file that you select to import was encrypted, you must enter the password that was used for encryption to enable decrypting sensitive connection properties\.

**A sample project from Samples** : You can create a project [from a project sample](https://dataplatform.cloud.ibm.com/samples?context=wx) to learn how to work with data in tools, such as notebooks to prepare data, analyze data, build and train models, and visualize analysis results\.

: The sample projects show how to accomplish goals, for example, to load and explore data, to create and train machine learning models for predictive analysis\. Each project includes the required assets, such as notebooks, and all the data sets that you need to complete the example use case\.

## Importing a project from a local file or sample ##

To import a project:

<!-- <ol> -->

1.  Click **New project** on the home page or on your **Projects** page\.
2.  Choose whether to create a project based on an exported project file or a sample project\.
3.  Upload a project file or select a sample project\.
4.  On the **New project** screen, add a name and optional description for the project\.
5.  If the project file that you select to import is encrypted, you must enter the password that was used for encryption to enable decrypting sensitive connection properties\. If you enter an incorrect password, the project file imports successfully, but sensitive connection properties are falsely decrypted\.
6.  Select the **Restrict who can be a collaborator** checkbox to restrict collaborators to members of your organization\. You can't change this setting after you create the project\.
7.  Choose an existing [object storage service instance](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/storage-options.html) or create a new one\.
8.  Click **Create**\. You can start adding resources if your project is empty, or begin working with the resources you imported\.

<!-- </ol> -->

## Learn more ##

<!-- <ul> -->

 *  [Administering a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/admin-project.html)
 *  [Exporting project](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/export-project.html)

<!-- </ul> -->

**Parent topic:**[Creating a project](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/projects.html)

<!-- </article "role="article" "> -->
# Viewing jobs across projects #

You can view the jobs that exist across projects for assets that run in tools, such as notebooks, Data Refinery flows, and SPSS Modeler flows\.

To view the status of jobs or job runs in projects:

<!-- <ol> -->

1.  From the navigation menu, select **Projects > Jobs**\.
2.  Select a view scope:
    
    <!-- <ul> -->
    
     *  **Jobs with finished runs**: all jobs that contain finished runs
     *  **Finished runs**: all job runs that have finished
     *  **Jobs with active runs**: all jobs that contain that contain active runs
     *  **Active runs**: all job runs that are still active
    
    <!-- </ul> -->
    
3.  Click ![the Filters icon](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/images/edit-filters.png) from the table toolbar to further narrow down the returned search results for the view scope you selected\. The filter options vary depending the view scope selection, for example, for jobs with active runs, you can filter by run state, job type and project, whereas for finished runs by time, run state, whether the runs were started manually or by a schedule, job type, run duration and project\.

<!-- </ol> -->

**Parent topic:**[Jobs](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/jobs.html)

<!-- </article "role="article" "> -->
# Leaving a project #

You can leave a project from within the project or from the **Projects** page\.

## Restrictions ##

If you are the only collaborator in the project with the **Admin** role, you must [assign the Admin role](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/collaborate.html) to another collaborator before you can leave the project\.

## Leaving a project from within the project ##

To leave a project from within the project:

<!-- <ol> -->

1.  Open the project\.
2.  On the **Manage** tab, go to the **General** page\.
3.  In the **Danger zone** section, click **Leave project**\.
4.  Click **Leave**\.

<!-- </ol> -->

## Leaving multiple projects ##

To leave one or more projects from the **Projects** page:

<!-- <ol> -->

1.  Select **View all projects** from the navigation menu\.
2.  Select one or more projects to leave\.
3.  Click **Leave**\.
4.  Click **Leave** to confirm\.

<!-- </ol> -->

**Parent topic:**[Projects ](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/manage-projects.html)

<!-- </article "role="article" "> -->
# Marking a project as sensitive #

When you create a project, you can mark the project as sensitive to prevent project collaborators from moving sensitive data out of the project\.

Marking a project as sensitive prevents collaborators of a project, including administrators, from downloading or exporting data assets, connections, or connected data from a project\. These sensitive assets cannot be added to a catalog or promoted to a space either\. Project collaborators with **Admin** or **Editor** role can export assets like notebooks or models from the project\.

When users open a project that is marked as sensitive, a notification is displayed stating that no data assets can be downloaded or exported from the project\.

## Restrictions ##

<!-- <ul> -->

 *  You cannot mark a project as sensitive after the project is created\.
 *  You cannot mark projects that use Git integration as sensitive\.

<!-- </ul> -->

**Parent topic:**[Administering projects](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/admin-project.html)

<!-- </article "role="article" "> -->
# Adding task credentials #

A task credential is a form of user authentication that is required by some services to perform operations in projects and spaces, for example to run certain tasks in a service or to enable the execution of long operations such as scheduled jobs without interruption\.

In IBM watsonx, IBM Cloud API keys are used as task credentials\. You can either provide an existing IBM Cloud API key, or you can generate a new key\. Only one task credential can be stored per user, per IBM Cloud account, and is stored securely in a vault\.

You can generate and rotate API keys in **Profile and settings > User API key**\.

Any user with an IBM Cloud account can create an API key\. The API key can be seen as a type of user name and password, enabling access to resources in your IBM Cloud account and should never be shared\.

If your service requires a task credential to perform an operation, you are prompted to provide it in the form of an existing or newly generated API key\.

Note that service administrators are responsible for defining a strategy to revoke task credentials when these are no longer required\.

## Learn more ##

<!-- <ul> -->

 *  [Managing the user API key](https://dataplatform.cloud.ibm.com/docs/content/wsj/admin/admin-apikeys.html)
 *  [Understanding API keys](https://cloud.ibm.com/docs/account?topic=account-manapikey&interface=ui)

<!-- </ul> -->

**Parent topic:**[Administering projects](https://dataplatform.cloud.ibm.com/docs/content/wsj/manage-data/manage-projects.html)

<!-- </article "role="article" "> -->
# Feedback logging in watsonx\.governance #

You can enable feedback logging in watsonx\.governance to configure model evaluations\.

To [manage feedback data](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-manage-feedback-data.html) for configuring quality and generative AI quality evaluations, watsonx\.governance must log your feedback data in the feedback logging table\.

Generative AI quality evaluations use feedback data to generate results for the following task types when you evaluate prompt templates:

<!-- <ul> -->

 *  Text summarization
 *  Content generation
 *  Question answering
 *  Entity extraction

<!-- </ul> -->

Quality evaluations use feedback data to generate results for text classification tasks\.

<!-- </article "role="article" "> -->
# Managing data for model evaluations in watsonx\.governance #

To enable model evaluations in watsonx\.governance, you must prepare your data for logging to generate insights\.

You must provide your model data to watsonx\.governance in a format that it supports to enable model evaluations\. watsonx\.governance processes your model transactions and logs the data in the watsonx\.governance data mart\. The data mart is the logging database that stores the data that is used for model evaluations\. The following sections describe the different types of data that watsonx\.governance logs for model evaluations:

## Payload data ##

Payload data contains the input and output transactions for your deployment\. To configure explainability and fairness and drift evaluations, watsonx\.governance must receive payload data from your model that it stores in a payload logging table\. The payload logging table contains the feature and prediction columns that exist in your training data and a prediction probability column that contains the model's confidence in the prediction that it provides\. The table also includes timestamp and ID columns to identify each scoring request that you send to watsonx\.governance as shown in the following example:

![Python SDK sample output of payload logging table](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/images/wos-ntbok.png)

You must send scoring requests to provide watsonx\.governance with a log of your model transactions\. For more information, see [Managing payload data](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-payload-logging.html)\.

## Feedback data ##

Feedback data is labeled data that matches the structure of training data and includes known model outcomes that are compared to your model predictions to measure the accuracy of your model\. watsonx\.governance uses feedback data to enable you to configure quality evaluations\. You must upload feedback data regularly to watsonx\.governance to continuously measure the accuracy of your model predictions\. For more information, see [Managing feedback data](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-manage-feedback-data.html)\.

## Learn more ##

[Sending model transactions](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-send-model-transactions.html)

<!-- </article "role="article" "> -->
# Managing feedback data for watsonx\.governance #

You must provide feedback data to watsonx\.governance to enable you to configure quality and generative AI quality evaluations and determine any changes in your model predictions\.

When you provide feedback data to watsonx\.governance, you can regularly evaluate the accuracy of your model predictions\.

## Feedback logging ##

watsonx\.governance stores the feedback data that you provide as records in a feedback logging table\.

The feedback logging table contains the following columns when you evaluate prompt templates:

<!-- <ul> -->

 *  **Required columns**:
    
    <!-- <ul> -->
    
     *  Prompt variable(s): Contains the values for the variables that are created for prompt templates
     *  `reference_output`: Contains the ground truth value
    
    <!-- </ul> -->
    
 *  **Optional columns**:
    
    <!-- <ul> -->
    
     *  `_original_prediction`: Contains the output that's generated by the foundation model
    
    <!-- </ul> -->
    
<!-- </ul> -->

## Uploading feedback data ##

You can use a feedback logging endpoint to upload data for quality evaluations\. You can also upload feedback data with a CSV file\. For more information, see [Sending model transactions](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-send-model-transactions.html)\.

## Learn more ##

[Sending model transactions](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-send-model-transactions.html)

**Parent topic:**[Managing data for model evaluations in Watson OpenScale](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-manage-data.html)

<!-- </article "role="article" "> -->
# Managing payload data for watsonx\.governance #

You must provide payload data to configure drift v2 and generative AI quality evaluations in watsonx\.governance\.

Payload data contains all of your model transactions\. You can log payload data with watsonx\.governance to enable evaluations\. To log payload data, watsonx\.governance must receive scoring requests\.

## Logging payload data ##

When you send a scoring request, watsonx\.governance processes your model transactions to enable model evaluations\. watsonx\.governance scores the data and stores it as records in a payload logging table within the watsonx\.governance data mart\.

The payload logging table contains the following columns when you evaluate prompt templates:

<!-- <ul> -->

 *  **Required columns**:
    
    <!-- <ul> -->
    
     *  Prompt variable(s): Contains the values for the variables that are created for prompt templates
     *  `generated_text`: Contains the output that's generated by the foundation model
    
    <!-- </ul> -->
    
 *  **Optional columns**:
    
    <!-- <ul> -->
    
     *  `input_token_count`: Contains the number of tokens in the input text
     *  `generated_token_count`: Contains the number of tokens in the generated text
     *  `prediction_probability`: Contains the aggregate value of log probabilities of generated tokens that represent the winning output
    
    <!-- </ul> -->
    
<!-- </ul> -->

The table can also include timestamp and ID columns to store your data as scoring records\.

You can view your payload logging table by accessing the database that you specified for the data mart or by using the [Watson OpenScale Python SDK](https://client-docs.aiopenscale.cloud.ibm.com/html/index.html) as shown in the following example:

![Python SDK sample output of payload logging table](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/images/wos-ntbok.png)

## Sending payload data ##

If you are using IBM Watson Machine Learning as your machine learning provider, watsonx\.governance automatically logs payload data when your model is scored\.

After you configure evaluations, you can also use a payload logging endpoint to send scoring requests to run on\-demand evaluations\. For production models, you can also upload payload data with a CSV file to send scoring requests\. For more information see, [Sending model transactions](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-send-model-transactions.html)\.

**Parent topic:**[Managing data for model evaluations in Watson OpenScale](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-manage-data.html)

<!-- </article "role="article" "> -->
# Configuring model evaluations in watsonx\.governance #

Configure watsonx\.governance evaluations to generate insights about your model performance\.

You can configure the following types of evaluations in watsonx\.governance:

<!-- <ul> -->

 *  [Quality](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-monitor-accuracy.html)  
    Evaluates how well your model predicts correct outcomes that match labeled test data.
 *  [Drift v2](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-driftv2-config.html)  
    Evaluates changes in your model output, the accuracy of your predictions, and the distribution of your input data
 *  [Generative AI quality](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-monitor-gen-quality.html)  
    Measures how well your foundation model performs tasks

<!-- </ul> -->

watsonx\.governance also enables [model health evaluations](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-model-health-metrics.html) by default to help you determine how efficiently your model deployment processes transactions\.

**Parent topic:**[Evaluating AI models with Watson OpenScale](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/getting-started.html)

<!-- </article "role="article" "> -->
# Payload logging in watsonx\.governance #

You can enable payload logging in watsonx\.governance to configure model evaluations\.

To [manage payload data](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-manage-payload-data.html) for configuring drift v2, generative AI quality, and model health evaluations, watsonx\.governance must log your payload data in the payload logging table\.

Generative AI quality evaluations use payload data to generate results for the following task types when you evaluate prompt templates:

<!-- <ul> -->

 *  Text summarization
 *  Content generation
 *  Question answering

<!-- </ul> -->

Drift v2 and model health evaluations use payload data to generate results for the following task types when you evaluate prompt templates:

<!-- <ul> -->

 *  Text classification
 *  Text summarization
 *  Content generation
 *  Entity extraction
 *  Question answering

<!-- </ul> -->

You can log your payload data with the payload logging endpoint or by uploading a CSV file\. For more information, see [Sending model transactions](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-send-model-transactions.html)

**Parent topic:**[Managing payload data in Watson OpenScale](https://dataplatform.cloud.ibm.com/docs/content/wsj/model/wos-manage-payload-data.html)

<!-- </article "role="article" "> -->