DataRobot-Research/drdocs huggingface dataset:
---
title: Target transform
description: How does transforming your target (log, ^2) help ML models and when should you use each?
---

# Target transform {: #target-transform }


<span style="color:red;font-size: 1rem"> `Robot 1`</span>

**How does transforming your target (`log(target)`, `target^2`, etc.) help ML models and when should you use each?**

<span style="color:red;font-size: 1rem"> `Robot 2`</span>


This is not going to be an ELI5 answer, but [here is a reason](https://www.codecademy.com/article/data-transformations-for-multiple-linear-regression){ target=_blank } you log transform your target.

TL;DR: When you run a linear regression, you try to approximate your response variable (e.g., target) by drawing a line through a bunch of points and you want the line to be as close to the response variable for those points as possible. Sometimes though, those points don’t follow a straight line. They might follow a curvy line, in which case a simple line through the points doesn’t approximate your target very well. In some of those scenarios, you can log transform your target to make the relationship between those points and your response variable more like a straight line. The following images from Code Academy show the bad:

![](images/rr-target-transform-1.png)

And the good:

![](images/rr-target-transform-2.png)

<span style="color:red;font-size: 1rem"> `Robot 3`</span>

This is specific to linear models that fit a straight line. For tree-based models like XGBoost, you don’t need to transform your target (or any other variable)!

<span style="color:red;font-size: 1rem"> `Robot 4`</span>

Yeah&mdash;log-transforming specifically was born out of trying to better meet the assumptions in linear regression (when they're violated). I have seen some cases where log-transformations can help from the predictive performance standpoint. (AKA when your target has a really long tail, log-transforming makes this tail smaller and this sometimes helps models understand the target better.)

<span style="color:red;font-size: 1rem"> `Robot 5`</span>

[Honey, I shrunk the target variable](https://florianwilhelm.info/2020/05/honey_i_shrunk_the_target_variable/){ target=_blank }.

<span style="color:red;font-size: 1rem"> `Robot 3`</span>

`Robot 4` you can also use a log link loss function (such as poisson loss) on both XGBoost and many linear regression solvers. I prefer that over the log transform, as the log transform biases the predicted mean, which makes your lift charts look funny on the scale of the original target.

But it really depends on the problem and what you’re trying to model!

<span style="color:red;font-size: 1rem"> `Robot 4`</span>

Or use inverse hyperbolic sine amirite? 😂

<span style="color:red;font-size: 1rem"> `Robot 1`</span>

Thanks all! Very helpful. `Robot 4` hyperbolic sine amirite is the name of my favorite metal band.
---
title: Specialized workflows
description: Leverage the support for alternative workflows for specialized data types such as anomaly detection, multilabel modeling, Visual and Location AI, and date/time partitioning.

---

# Specialized workflows {: #specialized-workflows }

The following sections describe alternative workflows for a variety of specialized data types:


Topic | Describes...
----- | ------
[Bias and Fairness](b-and-f/index) | Access an index page for quick links to all Bias and Fairness content.
[Composable ML](cml/index) | Build custom blueprints using built-in tasks and custom Python/R code.
[Location AI](location-ai/index) | Use geospatial analysis on spatial data.
[Unsupervised learning](unsupervised/index) | Work with unlabeled or partially labeled data to detect patterns, such as anomalies and clusters).
[Visual AI](visual-ai/index) | Apply visual learning to image data.
[Multilabel modeling](multilabel)| Perform modeling in which each row in a dataset is associated with one, several, or zero labels.
[OTV](otv) | Date/time partitioning for non-time series modeling.
[Text AI resources](textai-resources)  | Access links to the DataRobot Text AI functionality for working with text and viewing insights.
---
title: Defining redundant features
description: How does DataRobot define similarity in features and call them redundant?
---

# Defining redundant features {: #defining-redundant-features }

<span style="color:red;font-size: 1rem"> `Robot 1`</span>

**What makes a feature redundant?**

The [docs](feature-impact#remove-redundant-features-automl) say:

> If two features change predictions in a similar way, DataRobot recognizes them as correlated and identifies the feature with lower feature impact as redundant

 How do we quantify or measure "similar way"?

<span style="color:red;font-size: 1rem"> `Robot 2`</span>

If two features are highly correlated, the prediction difference (prediction before feature shuffle / prediction after feature shuffle) **of the two features should also be correlated**. The prediction difference can be used to evaluate pairwise feature correlation. For example, two highly correlated features are first selected. The feature with lower feature impact is identified as the redundant feature.

<span style="color:red;font-size: 1rem"> `Robot 1`</span>

Do we consider two features redundant when their prediction differences is the same/between `-x%` and `+x%`?

<span style="color:red;font-size: 1rem"> `Robot 2`</span>

We look at the correlation coefficient between the prediction differences and if it's above a certain threshold, we call the less important one (according to the models' feature impact) redundant.

<span style="color:red;font-size: 1rem"> `Robot 2`</span>

Specifically:

1. Calculate prediction difference before and after feature shuffle:

	`(pred_diff[i] = pred_before[i] - pred_after[i])`

2. Calculate pairwise feature correlation (top 50 features, according to model feature impact) based on `pred_diff`.

3. Identify redundant features (high correlation based on our threshold) then test that removal does not affect accuracy significantly.

<span style="color:red;font-size: 1rem"> `Robot 1`</span>

Thank you, Robot 2! Super helpful.
---
title: Version 8.0.x
description: DataRobot Release 8.0 release announcements index page.
---

# Version 8.0.x {: #version-80x }

Following are the AutoML, time series, and MLOps announcements for the features that make up DataRobot's 8.0.x Self-Managed AI Platform releases, first released _March 14, 2022_.

* [v8.0.x AutoML](v8.0.0-aml)
* [v8.0.x time series](v8.0.0-ats)
* [v8.0.x MLOps](v8.0-mlops)
* [Maintenance release notes](8.0-maintenance/index) for DataRobot v8.0.x.
---
title: V9.0.1
description: DataRobot Release 9.0.1 release notes.

---

# V9.0.1 {: #v901 }

_June 6, 2023_

The DataRobot v9.0.1 release includes some fixed issues in the DataRobot Self-Managed AI Platform platform. See the v9.0.0 release notes for:

* [Features introduced in v9.0.0](v9.0/index)

## Issues fixed in v9.0.1 {: #issues-fixed-in-v901 }

The following issues have been fixed since release v9.0.1:

### Data management {: #data-management }

- DM-5189: Supporting listing tables for Athena.
- DM-9894: Remove the **Method not supported** error when using treasure data JDBC driver.

### Predictions {: #predictions }

- CODEGEN-1670: Fixes an issue when detecting CSV line separators if the separator is `\r\n`.
- PRED-8594: Fixes an issue when setting the `columnNamesRemapping` property for a batch prediction job definition through a `PATCH` request.

### MLOps {: #mlops }

- AGENT-4155: Fixes an issue with force deletion of deployments in a Snowflake prediction environment.
- AGENT-4288: Updates the code snippet shown in external model deployments to not import a deprecated MLOps class (`OutputType`).
- AGENT-4253: Fixes an issue that prevented deployments created in the "default prediction environment" for enterprise installs from being listed in the prediction environment view.
- MMM-12255: Adds project and model information to User Activity Monitor entries for **Deployment Added** events
- MMM-12467: Fixes the histogram order for targets in external binary classification deployments created with holdout predictions.
- MMM-12742: Manually marks triggered retraining policy run as failed when encountering a credential error.
- MMM-12898: Fixes the target histogram for external deployments built inside DataRobot.
- MMM-13255: Fixes the calculation of "Accuracy" and "Balanced accuracy" metrics on the **Model Management Accuracy** tab for models with thresholds other than 0.5.
- MMM-12449: Fixes an issue when enabling feature drift for unsupervised model deployments.

### Time series {: #time-series }

- TIME-12787: Fixes partitioning set up for Segmentation to Clustering flow.
- TIME-12821: Removes misleading validation that restricted the insight computation to a max of 1000 forecast distance in absolute numbers.
- TIME-13090: Fixes the backtesting section of the **Advanced options** for OTV projects that have datasets larger than the time series limit.
- TIME-13098: Fixes an issue affecting holdout durations edition.

### Trust and Explainability {: #trust-and-explainability }

- TREX-3144: Fixes an issue when uploading a pairwise interactions CSV file in **Advanced options > Feature Constraints**.

_All product and company names are trademarks&trade; or registered&reg; trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them_.
---
title: Prepare external models
description: Prepare to create deployments from external models

---

# Prepare for external model deployment

External prediction environments and model packages allow you to deploy external (or remote) models to DataRobot. These models can make predictions on local infrastructure or any other external environment while DataRobot performs monitoring and management through the MLOps agents.

Topic | Describes
------|-----------
[Add an external prediction environment](pred-env) |  How to set up prediction environments on your own infrastructure, group prediction environments, and configure permissions and approval workflows.
[Manage external prediction environments](pred-env-manage) | How to edit, delete, and share external prediction environments, or deploy models to external prediction environments.
[Register external models](ext-model-reg) | How to register external models in the Model Registry.
[Manage external model packages](ext-model-manage) |  How to deploy, share, or archive external models from the Model Registry.
---
title: Offset/exposure with Gamma distributions
description: With DataRobot's monotonicity, to normalize or not to normalize, that is the question.
---

# Offset/exposure with Gamma distributions {: #offset-exposure-with-gamma-distributions }

<span style="color:red;font-size: 1rem"> `Robot 1`</span>

**How does DataRobot treat exposure and offset in model training with the target following a Gamma distribution?**

The target is total claim cost while `exposure = claim count`. So, in DataRobot, one can either set exposure equal to “claim count” _or_ set `offset = ln(“claim count”)`. Should I reasonably expect that both scenarios are mathematically equivalent?

Thanks you!

<span style="color:red;font-size: 1rem"> `Robot 2`</span>

Yes, they are mathematically equivalent. You either multiply by the exposure or add the `ln(exposure)`.

<span style="color:red;font-size: 1rem"> `Robot 1`</span>

Thanks, that was my impression as well. However, I did an experiment, setting up projects using the two approaches with the same feature list. One project seems to overpredict the target, while the other underpredicts. If they are mathematically equal, what might have caused the discrepancy?

<span style="color:red;font-size: 1rem"> `Robot 2`</span>

Odd.  Are you using the same error metric in both cases?

<span style="color:red;font-size: 1rem"> `Robot 1`</span>

Yes, both projects used the recommended metric&mdash;Gamma Deviance.

<span style="color:red;font-size: 1rem"> `Robot 2`</span>

Can you manually compare predictions and actuals by downloading the validation or holdout set predictions?

<span style="color:red;font-size: 1rem"> `Robot 1`</span>

Upon further checking, I see I used the wrong feature name (for the exposure feature) in the project with the exposure setting. After fixing that, predictions from both projects match (by downloading from the Predict tab).

I did notice, however, that the Lift Charts are different.

<span style="color:red;font-size: 1rem"> `Robot 2`</span>

That is likely a difference in how we calculate offset vs. exposure for Lift. I would encourage making your own Lift Charts in a notebook. Then you could use any method you want for handling weights, offset, and exposure in the Lift Chart.

<span style="color:red;font-size: 1rem"> `Robot 3`</span>

We do have a great AI Accelerator for [customizing lift charts](custom-lift-chart).

<span style="color:red;font-size: 1rem"> `Robot 1`</span>

Amazing. Thank you!
---
title: PCA and K-Means clustering
description: The impact of principal component analysis (PCA) on KMeans in DataRobot modeling
---

# PCA and K-Means clustering {: #pca-and-k-means-clustering }

<span style="color:red;font-size: 1rem"> `Robot 1`</span>

**What is the impact of principal component analysis (PCA) on K-Means clustering?**

Hi team, a customer is asking how exactly a PCA > k-means is being used during modeling. I see that we create a CLUSTER_ID feature in the transformed dataset and I am assuming that is from the k-means. My question is, if we are creating this feature, why aren't we tracking it in, for example, feature impact?


<span style="color:red;font-size: 1rem"> `Robot 2`</span>

Feature impact operates on the level of dataset features, not derived features. If we have one-hot encoding for categorical feature CAT1&mdash;we also calculate feature impact of just CAT1, not CAT1-Value1, CAT1-Value2,...

Permutation of original features would also produce permutation of KMeans results&mdash;so if those are important for the modeling result, its impact will be assigned to the original columns.

<span style="color:red;font-size: 1rem"> `Robot 3`</span>

Some blueprints use the one-hot-encoded cluster ID as features, and other blueprints use the cluster probabilities as features.

If you wish to assess the impact of the kmeans step on the outcome of the model, delete the kmeans branch in composable ML and use the Leaderboard to assess how the model changed.

As Robot 2 says, feature impact operates on the RAW data and is inclusive of both the preprocessing AND the modeling.
# Version 8.0.9 {: #version-809 }

_November 14, 2022_

The DataRobot v8.0.9 release includes some fixed issues in the DataRobot Self-Managed AI Platform platform. See the v8.0.0 release notes for:

* [Features introduced in v8.0.0](v8.0.0-aml)

In addition to the issues listed below, DataRobot addressed several outstanding security issues. This release/update is part of DataRobot's continual effort to secure the product and platform.

## Issues fixed in v8.0.9 {: #issues-fixed-in-v809 }

The following issues have been fixed since release v8.0.8:

### MLOps {: #mlops }

- MMM-11197: Fixed an issue where Bias and Fairness settings couldn't be configured if an external prediction environment was selected.

### Platform {: #platform }

- DM-8040: Fixed an issue in 8.0.8 when uploading parquet and avro files to the AI Catalog.

### Predictions {: #predictions }

- CODEGEN-1522: Time series Scoring Code now supports the 100th percentile for Prediction Intervals.
- PRED-8194: Increases the maximum age of the HTTP Strict Transport Secruity (HSTS) to 1 year on the Portable Prediction Server (PPS).

_All product and company names are trademarks&trade; or registered&reg; trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them_.
---
title: Relationship Quality Assessment speed improvements
description: Receive faster results from the Relationship Quality Assessment in Feature Discovery projects.
section_name: Data
maturity: public-preview
---

# Relationship Quality Assessment speed improvements {: relationship-quality-assessment-speed-improvements }

!!! info "Availability information"
    Speed improvements to the Relationship Quality Assessment, available as a public preview feature, is _on_ by default. Contact your DataRobot representative or administrator for information on disabling the feature.

    <b>Feature flag:</b> Enables Feature Discovery Relationship Quality Assessment Speedup

The [Relationship Quality Assessment (RQA)](fd-overview#relationship-quality-assessment) allows you to test relationships configurations in Feature Discovery projects by verifying join keys, dataset selection, and time-aware settings&mdash;the results of which help you spot and fix these configuration issues before EDA2 begins. Typically, finding the best configuration for a given use case requires multiple relationship iterations, however, current RQA run times are too long to make this feasible.

To improve run times, DataRobot now subsamples approximately 10% of the primary dataset, speeding up the computation without impacting the enrichment rate estimation accuracy or the results of the assessment. After the assessment is done, the sampling percentage is included at the top of the report.

![](images/rqa-speed-1.png)
---
title: Data preparation
description: How to add, profile, and wrangle data in Workbench.
---

# Data preparation {: #data-preparation }

{% include 'includes/data-description.md' %}

DataRobot’s wrangling capabilities give you the ability to prepare data and engineer features with a no-code interface to see transformations in real time&mdash;reducing the time from data to model.

This section covers the following topics:

Topic   |   Describes...
---------|-----------------
[Add data](wb-add-data/index)  | Add datasets to your Use Case from a local file, data connection, or the Data Registry.
[Wrangle data](wb-wrangle-data/index)  | Interactively prepare data for modeling without moving it from your data source to generate a new output dataset.
[Datasets tab](wb-data-tab)  | Manage the datasets linked to your Use Case and access Exploratory Data Insights.
[Reference](wb-data-ref) | View feature considerations and other reference material for Data Preparation.
---
title: Model log
description: How to display the model log, which shows the status of successful operations with green INFO tags and errors marked with red ERROR tags.

---

# Log {: #log }

The model log displays the status of successful operations with green INFO tags, along with information about errors marked with red ERROR tags. To display the model log, click a model on the Leaderboard list then click **Log**.

!!! note
    If you receive text-based insight model errors, see this [note](analyze-insights#text-based-insights) for a description of how DataRobot handles single-character "words."

![](images/model-log.png)
# MySQL {: #mysql }

## Supported authentication {: #supported-authentication }

- Username/password

## Prerequisites {: #prerequisites }

The following is required before connecting to MySQL in DataRobot:

- MySQL account

## Required parameters {: #required-parameters }

The table below lists the minimum required fields to establish a connection with MySQL:

Required field | Description |  Documentation
--------------- | ---------- |  -----------
`address`       | The service endpoint used to connect to MySQL.<br><br>**Example:**<br> `jdbc-cert-mysql.cqyt4ezythbf4.us-east-1.rds.amazonaws.com:3309`  |   [MySQL documentation](https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-reference-jdbc-url-format.html){ target=_blank }

For more information, see the [MySQL connector documentation](https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-reference.html){ target=_blank }.

{% include 'includes/data-conn-trouble.md' %}
---
title: Deployment settings
description: Add data to a deployment and configure monitoring, notifications, and challenger behavior using the Settings tab.

---

# Deployment settings {: #settings-tab }

!!! info "Deprecation notice"
        The **Settings > Data** and **Settings > Monitoring** tabs are deprecated and scheduled for removal. The new deployment settings workflow provides an organized and intuitive interface, separating the categories of deployment configuration and monitoring setup tasks into dedicated settings pages. During the deprecation period, you can use the **Data** tab; however, the **Monitoring** tab directs you to the [service health settings](service-health-settings).

You can add data to a deployment and configure monitoring, notifications, and challenger behavior using the **Settings** associated with each deployment tab:

 Topic | Describes
-------|------------
[Set up service health monitoring](service-health-settings) | Enable [segmented analysis](deploy-segment) to assess service health, data drift, and accuracy statistics by filtering them into unique segment attributes and values. 
[Set up data drift monitoring](data-drift-settings)         | Enable [data drift monitoring](data-drift) on a deployment's Data Drift Settings tab. 
[Set up accuracy monitoring](accuracy-settings)             | Enable [accuracy monitoring](deploy-accuracy) on a deployment's Accuracy Settings tab.
[Set up fairness monitoring](fairness-settings)             | Enable [fairness monitoring](mlops-fairness) on a deployment's Fairness Settings tab. 
[Set up humility rules](humility-settings)                  | Enable [humility monitoring](humble) by creating rules which enable models to recognize, in real-time, when they make uncertain predictions or receive data they have not seen before.
[Configure retraining](retraining-settings)                 | Enable [Automated Retraining](set-up-auto-retraining) for a deployment by defining the general retraining settings and then creating retraining policies.
[Configure challengers](challengers-settings)               | Enable [challenger comparison](challengers) by configuring a deployment to store prediction request data at the row level and replay predictions on a schedule.
[Review predictions settings](predictions-settings)         | Review the Predictions Settings tab to view details about your deployment's inference data.
[Enable data export](data-export-settings)                  | Enable [data export](data-export) to compute and monitor custom business or performance metrics. 
[Set prediction intervals for time series deployments](predictions-settings#set-prediction-intervals-for-time-series-deployments) | Enable [prediction intervals](ts-predictions#prediction-preview) in the prediction response for deployed time series models.