---
title: Feature Discovery projects
description: How to create a project from multiple datasets. You define the relationships. Feature Discovery aggregates the secondary datasets to enrich the primary dataset.

---

# Feature Discovery projects {: #feature-discovery-projects }

Feature Discovery is based on relationships&mdash;between datasets and the features within those datasets. DataRobot provides an intuitive relationship editor that allows you to build and visualize these relationships. The end product is a multitude of additional features that result from these linkages. These derived features can then train more accurate models and generate better predictions. DataRobot’s Feature Discovery engine analyzes the graphs and the included datasets to determine a feature engineering “recipe,” and from that recipe generates secondary features for training and predictions.

!!! note
    See the Feature Discovery [file requirements](file-types#feature-discovery-file-import-sizes) for dataset sizes information.

Review the next section to [get started with Feature Discovery](#get-started-with-feature-discovery). Or, skip to the step-by-step instructions that describe:

1. [Add datasets to a project](#add-datasets).
2. [Create relationships](#define-relationships).
3. [Set join conditions](#set-join-conditions).
4. [Assess the quality of relationship configurations](#relationship-quality-assessment).
5. [Configure Feature Discovery settings](#feature-discovery-settings).
6. [Start the project](#start-the-project).

You can also take a deeper dive into [time-aware feature engineering](fd-time), [derived features](fd-gen), and [making predictions](fd-predict) on models that have derived features.

## Get started with Feature Discovery {: #get-started-with-feature-discovery }

In most cases, all you need to start a Feature Discovery project is a simple *primary* dataset that includes:

* The target (column that you want to predict).
* An identifier (for example, *customer_id* or *transaction_id*) to link the dataset to additional related datasets. This *key* serves as the basis of dataset joins.
* An optional time index&mdash;a date feature in the primary dataset&mdash;to support [time-aware Feature Discovery](fd-time#prediction-point-and-time-indexes). This date feature is used as the prediction point for generating new features.

Each record of the primary dataset represents the desired unit of analysis. From this primary dataset, DataRobot guides you through creating relationships to additional datasets, called *secondary datasets*.

![](images/safer-1.png)

Secondary datasets have features that can potentially enrich the primary dataset. While it may be the case that both primary and secondary datasets have one-to-one relationships when they are added, it is not required. In most cases, DataRobot aggregates and then summarizes features in the secondary datasets, and, from there, enriches the primary dataset.

### Sample use case {: #sample-use-case }

The following sections use an example to illustrate how DataRobot automatically discovers new features from multiple datasets to predict whether a loan will default. In the primary dataset, **CreditRisk - Loan Applications**, the *is-bad* column is the project target. The relation between the datasets is the *CustID* column.

![](images/safer-primary-dataset.png)

Two additional relational datasets, **CreditRisk - Credit Inquiries** and **CreditRisk - Tradeline Acccounts**, are the secondary datasets used for Feature Discovery.

![](images/safer-secondary-datasets.png)

Once model building begins, DataRobot runs through EDA2, adding newly created features to the [**Data** page](fd-gen). The **Data** page provides a variety of information about all the resulting project data, both new and old.

##  Add datasets {: #add-datasets }

From the **AI Catalog**, select the primary dataset and click **Create project**. Then, enter the target feature.

!!! note
     This procedure shows how to load datasets using the **AI Catalog**, so to begin, make sure all the assets are in the catalog. Alternatively, you can use the drag-and-drop method to upload datasets. If you do so, all datasets that you upload are automatically registered to the **AI Catalog**.

_A valid Feature Discovery project requires at least one secondary dataset_&mdash;the following tabs describe how to load additional datasets into the project from both the **Start** page and the **relationship editor**:

=== "From the Start page"

    1. On the **Start** page, click **Add datasets** to add one or more _additional_ datasets to the project.

        ![](images/safer-secondary-add-datasets.png)

    3. On the **Specify prediction point** page of the **relationship editor**, optionally **Select a date feature to use as a prediction point**. This date/time feature from the primary dataset serves as a reference date for feature derivation windows.

        ![](images/safer-rel-editor-select-date.png)

        !!! note
    	     The step to specify a prediction point does not display if you have already specified a prediction point for the project.

           For an in-app explanation of prediction points, expand **Show Example**.

    4. Click **Set up as prediction point** for a time-aware Feature Discovery project or **Continue without prediction point** for a non time-aware project.

        !!! note
            Although you can select the same date feature used for the out-of-time validation (OTV) partition as the prediction point, clicking **Continue without prediction point** automatically uses the OTV partition feature when generating new features.

            If you [add or edit the prediction point](#primary-datasets), DataRobot accounts for that change when generating new features. 

    5. In the **Add datasets** page of the **relationship editor**, select a data import method under **Add Data From**.

        ![](images/safer-rel-editor-add-data-from.png)

    	 This example shows how to add a dataset from the **AI Catalog**.

    6. From the **AI Catalog**, select the datasets you want to include by clicking **Select**. Use the search functionality to easily locate datasets for selection. When finished, click **Add**.

    	 ![](images/safer-rel-editor-select-datasets.png)

    7. Click **Continue** to finalize your selection. The secondary datasets you select on this page are immediately added to the configuration, so if you reload the page without clicking **Continue**, the data is not lost.

        ![](images/safer-rel-editor-continue.png)

        The **Define Relationships** page displays the datasets.

        ![](images/safer-define-relationships-datasets.png)

    Best practice suggests continuing within this editor to define relationships. You can, however, click **Continue to project** to return to the **Start** screen.

    ![](images/safer-secondary-datasets-on-start-page.png)

    The datasets display and you can see the number of relationships that have been defined.

    At any time, you can click **Define relationships** to return to the **Define Relationships** page.

=== "From the relationship editor"

    If your project has more than one secondary dataset, you can add more datasets after saving. From the **Define Relationships** page:

    1. Click **Add datasets** and select a data import method.

        ![](images/safer-add-data-1.png)

        This example shows how to add a dataset from the **AI Catalog**.

    2. From the **AI Catalog**, select the datasets you want to include by clicking **Select**. Use the search functionality to easily locate datasets for selection. When finished, click **Add**.

        ![](images/safer-add-data-2.png)

        The **Define Relationships** page displays the datasets.

        ![](images/safer-add-data-3.png)

Each dataset displayed on the canvas has a menu with shortcuts to dataset-related tasks. See details of working with [primary datasets](#primary-datasets) and [secondary datasets](#secondary-datasets).

After adding secondary datasets to your project, [define the relationships](#define-relationships) between the datasets.

### Snowflake integration {: #snowflake-integration }

An integration between DataRobot and Snowflake allows joint users to both execute data science projects in DataRobot and perform computations in Snowflake as a way to optimize workload performance. **Feature Discovery** training and prediction workflows will push down relational inner-joins, projection, and filter operations to the Snowflake platform (via SQL). By natively conducting joins in the Snowflake database, data is filtered into smaller datasets for transfer across the network before loading into DataRobot. The smaller datasets reduce project runtimes.

To enable integration with Snowflake, the following requirements must be met:

 * A Snowflake [data connection](data-conn#dataconn-add) is set up.
 * All secondary datasets are stored in Snowflake.
 * All Snowflake sources are stored in the same warehouse.
 * All datasets are configured as [dynamic datasets](catalog-asset#asset-states) in the AI Catalog.
 * You have write permissions to one of the schemas in use or one `PUBLIC` schema of the database in use.

If the above requirements are met, DataRobot automatically establishes the integration and displays the Snowflake icon and **Snowflake mode enabled**, in blue, at the top of the **Define Relationships** page.

![](images/snowflake-safer-1.png)

### View dataset details {: #view-dataset-details }

You can access dataset details directly from the relationship editor using one of the following methods:

=== "Brief description"

    On the dataset tile, hover over the line beneath the dataset name to display metadata for the dataset.

    ![](images/safer-dataset-brief-details.png)


=== "Detailed description"

    Click the menu icon on the top right of the dataset tile and select **Details** to open the [**Info**](catalog-asset#work-with-metadata) page in the **AI Catalog**. From here you can access the profile, feature lists, relationships, version history, and comments associated with the dataset.

    ![](images/safer-dataset-details-menu-item.png)

    You can also delete the dataset from this menu.

    ![](images/safer-define-relationships-menu.png)

##  Define relationships {: #define-relationships }

Once all datasets are loaded, the next step is to define relationships on the **Define Relationships** page. The primary dataset is on the canvas while any secondary sets are listed in the left pane. After establishing a relationship between two datasets, you can define the relationship by setting [join conditions](#set-join-conditions) and [feature derivation windows (FDW)](#set-feature-derivation-windows) for time-aware feature engineering.

To define relationships:

1. Click a secondary dataset to highlight it; notice the addition of a plus sign on the primary set.

    ![](images/safer-define-relationships-plus.png)

2. Click the plus sign. DataRobot adds the selected secondary dataset to the canvas and opens the configuration editor.

	 ![](images/safer-8e.png)

    The following table describes the elements of the **Create new relationship** page:

    |   |  Element  |  Description |
    |---|-----------|--------------|
    | ![](images/icon-1.png) | Secondary dataset for join  | Sets the secondary dataset used in the join. Change via the dropdown to any added dataset. Changes are reflected in the canvas below.  |
    | ![](images/icon-2.png)  | Primary dataset for join | Sets the primary dataset used in the join.  |
    | ![](images/icon-3.png) |  Suggested join condition  |  Sets the join condition (feature) for the corresponding dataset (listed above the condition). DataRobot suggests up to five conditions, each of which is editable. Use the dropdown to select a new feature; use the trash icon (![](images/icon-trash.png)) to delete the join.  |
    | ![](images/icon-4.png) |  Add join condition |  Provides a manual join configuration option. |
    | ![](images/icon-5.png) | Save or Save and configure time-aware  | Saves the relationship configuration. **Save** is the option if there is no date feature or you did not set a prediction point. If you did set a [prediction point](#load-datasets) from the primary dataset, the **Save and configure time-aware** button displays.   |
    | ![](images/icon-6.png) | Canvas display controls   |  Zooms in or out, or resets the default display size.  |
    | ![](images/icon-7.png) |  Dataset menu options   | Provides access to a variety of actions that can be enacted on a [primary](#primary-dataset-menu-actions) or [secondary](#secondary-dataset-menu-actions) dataset. |
    | ![](images/icon-8.png) |  Join edit launch  |  Opens the relationship editor, allowing you to define or modify the relationship between the datasets joined by the line you clicked. |
    | ![](images/icon-9.png) |  Primary icon  |  Indicates, with a bullseye icon, that this is the primary dataset. |
    | ![](images/icon-10.png) |  Tour launch  |  Opens a short tour that provides an overview of configuring Feature Discovery. |
    | ![](images/icon-11.png) | Continue to project |  Returns to the **Start** screen where you can revise your time-aware settings, set advanced options, set a modeling mode, and start the modeling process.|

###  Set join conditions {: #set-join-conditions }

If tables in your datasets are well-formed, DataRobot automatically detects compatible features and creates up to five "suggested" joins. You can modify the suggested join using the dropdowns associated with each join key.

![](images/safer-8f.png)

You can also manually create join keys by clicking **Add join condition**. In the resulting dialog, select a join feature from each dataset from the feature dropdown.

??? note "Join feature type compatibility and restrictions"
    See the table below for compatible join types when creating or modifying joins in the relationship editor:

    Feature type       | Compatible join types
    ---------- | -----------
    Numeric      | Numeric, Categorical
    Categorical      | Categorical, Numeric, Text
    Text      | Text, Categorical
    Date      | Date

    The following feature types cannot be used as join keys:

    * Summarized Categorical
    * Length
    * Currency
    * Percentage
    * Audio
    * Image
    * Document

![](images/safer-8g.png)

Once you've added all of your secondary datasets and selected your relationship configuration settings, click **Save and configure time-aware** or **Save** for a non time-aware project.

* If the project is not time-aware, the **Start** page displays.
* If the project is time-aware, the **Time-aware feature engineering** page displays where you can [configure FDWs](#set-feature-derivation-windows).

### Set feature derivation windows {: #set-feature-derivation-windows}

After adding secondary datasets to a time-aware project, you can define the FDWs&mdash;a rolling window of past values used to generate features before the prediction point. The FDW constrains the time history&mdash;in the example below, no further back than 30 days, no more recent than 2 days.

1. Click **Select time feature** to choose a time index feature for the secondary dataset.

    ![](images/safer-time-aware-feature.png)

2. Configure the FDWs. You can configure up to three FDWs for each dataset, but each window must be unique. To add a FDW, click **Add window**.

    ![](images/safer-time-aware-set-fdw.png)

    Once set, the FDW is reflected in the dataset's tile on the canvas:

    ![](images/safer-time-aware-fdw.png)

    These time-aware settings ensure that the generated features are based only on records that occur before the prediction point. For more details, see [Time-aware feature engineering](fd-time).

## Work with datasets {: #work-with-datasets }

Once a dataset is added to the canvas, you can modify and refine its configuration. Primary datasets appear on the canvas by default, but all secondary datasets must be added.

###  Primary datasets {: #primary-datasets }

!!! note
    Be sure to save your configuration before using the menu options. Unsaved changes are lost when you leave a page.

Working from the canvas, you can select the menu option on the dataset tile. The primary dataset allows you to add a relationship or edit the prediction point:

![](images/safer-prim-menu.png)

|  Option   |   Description  |
|-----------|----------------|
|  Add relation  |  Choose **Add relation** when you don't have any previous relationships configured to open the **Create new relationship** page. This is the equivalent of selecting the dataset from the list on the left and clicking the plus sign on the primary's canvas tile. Once the page opens, select a secondary dataset from the dropdown and it is added to the canvas. |
|  Edit prediction point   | Select **Edit prediction point** to choose a different date feature to use as your prediction point. |

### Secondary datasets {: #secondary-datasets }

When a secondary dataset has been selected and moved to the canvas, a menu option becomes available on its tile. The table below describes the options available from the menu:

![](images/safer-8j.png)

|  Option   |   Description  |
|-----------|----------------|
|  Add relation  |  Opens the relationship editor and allows you to select a dataset (from any available in the left pane) to join with. |
|  Edit alias   | Allows you to set an alias for the dataset. The string displays on the canvas as the secondary dataset name. The alias does not change the display in the left-pane dataset list or the relationship editor pages. |
|  [Configure dataset](#configure-secondary-datasets)  |  Opens the dataset configuration editor, where you can set dataset details.  |
|   Configure time-awareness  | Opens the time-aware feature engineering configuration dialog, where you can select a time index for the secondary dataset or confirm that the correct date/time feature is selected.   |
|  Details   |  Click to open the [**Info**](catalog-asset#work-with-metadata) window for the dataset in the **AI Catalog**.   |
|  Delete   |  Deletes the dataset, and all its relationships, from the current relationship configuration. The dataset is still available to the configuration and listed in the left panel.  |

#### Configure secondary datasets {: #configure-secondary-datasets }

Selecting **Configure dataset** from a secondary dataset menu opens the **Dataset Editor**.

![](images/safer-dataset-editor.png)

From here you can:

* Change the dataset alias. If not manually set, DataRobot auto-generates an alias based on the file name. Click in the box to modify the alias; the alias for the primary dataset cannot be modified.

* Choose a snapshot policy, either Latest, Fixed, or Dynamic, to use for this project. By default, the selected snapshot policy will apply at [prediction time](fd-predict).

* Choose a feature list to apply against the corresponding dataset. Use this option to limit the size of the table by selecting relevant features. You can create new feature lists from the [**AI Catalog**](catalog-asset#work-with-feature-lists).

## Relationship Quality Assessment {: #relationship-quality-assessment }

After configuring at least one secondary dataset, you can test the quality of those relationship configurations to identify and resolve potential problems early in the creation process. The Relationship Quality Assessment tool verifies join keys, dataset selection, and time-aware settings before EDA2 begins.

Click the **Review configuration** button to trigger the Relationship Quality Assessment.

![](images/rqa-1.png)

A progress indicator (loading spinner) displays on each dataset and on the **Review Configuration** button, which is disabled, to indicate that an assessment is currently running.

![](images/rqa-2.png)

Once the assessment is complete, DataRobot marks all tested datasets. Those with identified issues display a yellow warning icon and those with no identified issues display a green tick.

??? tip "Deep dive: Relationship assessments"
    Depending on the project type, DataRobot assesses the relationship's enrichment rate, window settings, and most recent data&mdash;each of which is described in the table below:

    | Category       | Description | Solution | Project type |
    | ---------- | ----------- | -------- | --------- |
    | Enrichment rate      | Quickly determines, as a percentage, how many rows in the secondary dataset map to rows in the primary table.    | Review the dataset and relationship. | All        |
    | Window settings      | Determines how many rows in the secondary dataset map to the primary dataset within the specified FDWs.       | Expand the window settings to find more rows. | Time-aware        |
    | Most recent data      | Compares the minimum and maximum time index of the secondary and primary datasets to determine if the secondary dataset is outdated.     | Review the selected feature list and snapshot policy. | Time-aware        |

    Assessments are always updated for JDBC sources with dynamic snapshot policy.

    DataRobot calculates enrichment rate using the following formula:

    (`rows_of_primary_that_can_be_mapped_to_secondary` / `total_rows_of_primary`) x `100`

Select the warning icon to view a summary of the issues with suggested potential fixes. A summary of the issues identified during the assessment is displayed at the top of the window.

![](images/rqa-3.png)

To open the detailed report, click the orange arrow on the right. DataRobot breaks down the assessment by category, providing additional information to diagnose the issue. If a secondary dataset has multiple FDWs, a detailed report is created for each one.

![](images/rqa-4.png)

To resolve warnings, click the orange link displayed below each warning&mdash; Review dataset, Review relationship, or Review window settings&mdash;and a pane appears at the top of the relationship editor allowing you to modify relationship configurations.

![](images/rqa-5.png)

After EDA2 completes and model building begins, you can view the most recent Relationship Quality Assessment in the **Data > Feature Discovery** tab.

## Feature Discovery settings {: #feature-discovery-settings}

The Feature Discovery process uses a variety of heuristics to determine the list of [features to derive](fd-gen) in a DataRobot project. In **Feature Discovery Settings**, you can control which transformations DataRobot will try when deriving new features ([feature engineering controls](#feature-engineering-controls)), as well as set DataRobot to automatically remove redundant features and those with low impact ([feature reduction](#feature-reduction)).

To access **Feature Discovery Settings**, click the settings gear on the **Define Relationships** page.

![](images/safer-relationship-configuration-settings.png)

### Feature engineering controls {: #feature-engineering-controls }

You can influence how DataRobot conducts feature engineering by setting feature engineering controls. You might want to do this to:

* Use your domain knowledge to guide the feature engineering process and improve the quality of the derived features.
* Speed up feature engineering.
* Improve accuracy by deriving more features, for example, using [categorical statistics](fd-gen#categorical-statistics), skewness, and kurtosis.
* Exclude specific transforms that might be too complex to explain to business stakeholders. You can exclude these features post-modeling but that adds to the complexity of the modeling process.

Set the feature engineering options in the relationship editor prior to [EDA2](eda-explained#eda2).

2. In **Feature Discovery Settings**, click the **Feature Engineering** tab. Consider which feature engineering transformations make the most sense for your project and select the ones you want DataRobot to try when deriving new features.

    ![](images/safer-feature-engineering-controls.png)

    You can hover over a transformation to view a tool tip that describes it.

    ![](images/safer-feature-engineering-control-tool-tip.png)

    ??? note "Latest vs. Latest within window"

        Transformation       | Description | Default
        ---------- | ----------- |
        Latest      | Generates new features by exploring all historical data up until the end point of any defined FDWs. Note that this method ignores all FDW start points.  | Disabled
        Latest within window      | Generates new features within the defined FDW. For time-aware feature engineering, only the data within the FDW is required when making predictions. | Enabled


3. Click **Save changes**.

### Feature reduction {: #feature-reduction }

During Feature Discovery, DataRobot generates new features then removes the features that have low impact or are redundant. This is called *feature reduction*. You can instead include all features when building models by disabling feature reduction using the following method:

In the relationship configuration (the **Define Relationships** page), click the settings (![](images/icon-gear.png)) gear. Select the **Feature Reduction** tab and toggle off **Use supervised feature reduction**:

![](images/safer-settings-feature-reduction.png)

##  Start the project {: #start-the-project }

1. Once you are happy with the definition of the relationship(s), click **Continue to project** to return to the **Start** screen.

    ![](images/safer-secondary-datasets-done.png)

    The **Secondary Datasets** section provides visual queues that provide details about the secondary datasets.

    | | Visual queue | Indicates
    |---|---|---|
    | ![](images/icon-1.png) | Datasets with blue text | The dataset is in use and part of the project. |
    | ![](images/icon-2.png) | Datasets with white text | The dataset is loaded but not part of the relationship definition. |
    | ![](images/icon-3.png) | Linked datasets | The number of datasets linked with this dataset. |
    | ![](images/icon-4.png) | Number of datasets and relationships | The number of secondary datasets and how many have relationships defined. |

2. Click **Start**.

    DataRobot conducts feature engineering as part of EDA2 and begins generating model blueprints.

### Share assets {: #share-assets }

As with any DataRobot project, you can share Feature Discovery projects (depending on your permissions). The assignable [roles](roles-permissions) provide different levels of permission for the recipient. Unique to Feature Discovery projects, however, is the ability to share engineering graphs and datasets as well.

To share a project, click the share icon (![](images/icon-share.png)). For the recipient to interact with the project, they must have access to the additional assets. By default, assets are not shared. Check to enable sharing relationships and datasets, or DataRobot provides a warning:

![](images/safer-34.png)

Note that in addition to the assigned role, the listing of project users also indicates whether project assets have been shared.

![](images/safer-35.png)