DataRobot-Research/drdocs huggingface dataset:
---
title: Migrate a model to a new cluster
description: Download a deployed model from DataRobot cluster X, upload it to DataRobot cluster Y, and then deploy and make requests from it.

---

# Migrate a model to a new cluster {: #migrate-a-model-to-a-new-cluster}

[Access this AI accelerator on GitHub <span style="vertical-align: sub">:material-arrow-right-circle:{.lg }</span>](https://github.com/datarobot-community/ai-accelerators/blob/main/advanced-experimentation/model_migration/Model_Migration_Example.ipynb){ .md-button }

Currently under development, an experimental DataRobot API allows administrators to download a deployed model from DataRobot cluster X, upload it to DataRobot cluster Y, and then deploy and make requests from it.

Note that this notebook will not work using https://app.datarobot.com.

### Prerequisites

* This notebook must be able to write to the model directory, located in the same directory as this accelerator's notebook. For best results, run this notebook from the local file system
* Ensure that the model you choose to migrate must be a deployed model.
* Provide API keys for both the source and destination clusters.
* The Source and Destination users must have the "Enable Experimental API access" feature flag enabled to follow this workflow.
* The notebook must have connectivity to the Source and Destination clusters.
* DataRobot versions on the clusters must be consistent with the Supported Paths above.
* For models on clusters of DataRobot v7.x, you must have SSH access to the App Node of the cluster.
* The Source and Destination DataRobot clusters must have the following in the config.yaml:
---
title: Data registry
description: The Data Registry lists all static and snapshot datasets you currently have access to in the AI Catalog, including those uploaded from local files and data connections in Workbench.

---

# Data registry {: #data-registry }

When you open the **Add data** modal, by default, DataRobot displays the **Data Registry**, a central catalog for your datasets that lists all static and snapshot datasets you currently have access to in the AI Catalog, including those uploaded from [local files](wb-local-file) and [data connections](wb-connect) in Workbench. This method of adding data is a good approach if your dataset is already ready for modeling.

When you add a dataset from the registry, you're creating a link from the Use Case to the source of that dataset, meaning datasets can have a one-to-many relationship with Use Cases, so when a dataset is removed, you're only removing the link; any experiments created from the dataset will not be affected.


![](images/wb-datareg-1.png)

See the associated [considerations](wb-data-ref/index#add-data) for important additional information.

## Add a dataset {: #add-a-dataset }

You can add any static or snapshot datasets that have been previously registered in DataRobot.

To add a dataset:

1. In the **Data Registry**, select the box to the left of the dataset you want to view.

    ![](images/wb-datareg-6.png)

2. (Optional) [Preview the dataset](#preview-a-dataset) to determine if the dataset is appropriate for the objective of your Use Case by clicking **Preview**.
3. Click **Add to Use Case** in the upper-right corner.

    ![](images/wb-datareg-7.png)

    Workbench opens to the **Datasets** tab of your Use Case.

## Preview a dataset {: #preview-a-dataset}

Viewing a snapshot preview allows you to confirm that a dataset is appropriate for your Use Case before adding it.

To preview a dataset:

1. In the **Data Registry**, select the box to the left of the dataset you want to view and click **Preview** in the upper-right corner.

    ![](images/wb-datareg-2.png)

2. Analyze the dataset using the **Features** and **Data preview** buttons:

    === "Features"

        Lists the feature name, type, number of unique values, and number of missing values for each feature in the dataset.

        ![](images/wb-datareg-4.png)

    === "Data preview"

        Displays a random sample, up to 1MB, of the raw data table.

        ![](images/wb-datareg-5.png)

3. Determine if the dataset suits your Use Case, and then either:

      - Add the dataset to your Use Case by clicking **Add to Use Case**.
      - Go back to the Data Registry by clicking **Data Registry** in the breadcrumbs below the dataset name.

## Next steps {: #next-steps }

From here, you can:

- [Add more data.](wb-add-data/index)
- [View Exploratory Data Insights for the dataset.](wb-data-tab)
- [Use the dataset to set up an experiment and start modeling.](wb-experiment/index)


## Read more {: #read-more}

To learn more about the topics discussed on this page, see:

- [Asset states in DataRobot.](catalog-asset#asset-states){ target=_blank }
---
title: Text AI resources
description: Provides links to Text AI resources available in DataRobot.

---

# Text AI resources {: #text-ai-resources }

Text AI in DataRobot allows you to seamlessly incorporate text data into your model without being a Natural Language Processing (NLP) expert and without injecting extra steps in the model building process. With models and preprocessing steps designed specifically for NLP, DataRobot supports _all languages_ from [ISO 639](https://en.wikipedia.org/wiki/List_of_ISO_639-2_codes){ target=_blank }, the set of standards for representing names for languages and language groups.

The tools available for working with text are described in the following sections.

Topic | Describes...
---|---
**Working with text** | :~~:
[Automated transformations](model-ref#automated-feature-transformations) | Learn about DataRobot's automated feature engineering for text, built to enhance model accuracy.
[Clustering based on text collections](clustering) | Use clustering for detecting topics, types, taxonomies, and languages in a text collection.
[Aggregation and imputation in time series projects](ts-data-prep#set-manual-options) | Set handling for text features in time series projects.
[Composable ML transformers](cml-blueprint-edit) | Edit model blueprints, including pre-trained transformers, to best represent text features.
**Model insights** | :~~:
[Coefficients](coefficients)  | See how text-preprocessing transforms text found in a dataset into a form that can be used by a DataRobot model.
[Text Mining](analyze-insights#text-mining) | Display the most relevant words and short phrases in any variables detected as text.
[Word cloud](analyze-insights#word-clouds) | Display the most relevant words and short phrases found in your dataset in word cloud-format.
[Text Explanations](predex-text) | Visualize not only the text feature that is impactful, but also which specific words within a feature are impactful.
[Multilabel modeling for text categorization](multilabel) | Use multilabel classification for text categorization.
[Example: Capturing sentiment in text (link not live yet)](cml-ref/index) | See an example of uplifting a model by capturing sentiment in the text.
**Text-related feature announcements** | :~~:
[NLP Fine-Tuner blueprints](v8.0.0-aml#nlp-fine-tuner-blueprints-for-multi-modal-datasets-in-any-langauge) | Read about NLP Fine-Tuner blueprints.
[FastText for language detection](july2022-announce#nlp-autopilot-with-better-language-support-now-ga) | Read about FastText for language detection at data ingest.
[TinyBERT featurizer](v7.1.0-aml#tiny-bert-pretrained-featurizer-implementation-extends-nlp) | Read about using Google's Bidirectional Encoder Representations from Transformers (distilled version).
---
title: REST API code examples
description: Review comprehensive workflows, notebooks, and tutorials that help you find complete examples of common data science and machine learning workflows.

---

# REST API code examples {: #rest-api-code-examples }

The API user guide includes overviews and workflows for DataRobot's REST API that outline complete examples of common data science and machine learning workflows. Be sure to review the [API quickstart guide](api-quickstart/index) before using the notebooks below.

Topic | Describes... |
----- | ------ |
[Create a multiseries project](multi-rest.ipynb) | How to initiate a DataRobot project for a multiseries time series problem using the DataRobot REST API.
[Create a clustering project](rest-cluster.ipynb) | How to create a clustering project and initiate Autopilot in Manual mode via DataRobot's REST API. |
[Fetch metadata from prediction jobs](pred-metadata.ipynb) | How to retrieve metadata from prediction jobs with DataRobot's REST API. |
---
title: Batch prediction UI
description: Use a deployment's batch prediction interface to score large files efficiently.
---

# Batch prediction UI {: #batch-scoring-methods }

To make batch predictions from the UI, you must first deploy a model. After deploying, navigate to the [**Make Predictions** tab](batch-pred) and use the interface to either make a [one-time batch prediction](batch-pred) or [configure batch prediction jobs](batch-pred-jobs).

You can also view [example configurations using a Snowflake database](pred-job-examples-snowflake).
There are several options available in the **Actions** menu, which can be accessed for each model package in the **Model Packages** tab of the **Model Registry**:

![](images/reg-action-1.png)

The available options depend on a variety of criteria, including user permissions and the data available to your model package:

![](images/reg-action-2.png)

Option | Description
-------|------------
Deploy | Select **+ Deploy** to [create a deployment](deploy-model#deploy-from-the-model-registry) from a model package. For external models, you can [create an external deployment](deploy-external-model#deploy-an-external-model-package).
Share  | The sharing capability allows [appropriate user roles](roles-permissions#reg-roles) to grant permissions on a model package. To share a model package, select the **Share** (![](images/icon-share.png)) action. <br> You can only share up to your own access level (a consumer cannot grant an editor role, for example) and you cannot downgrade the access of a collaborator with a higher access level than your own.    
Permanently Archive | If you have the appropriate [permissions](roles-permissions#reg-roles), you can select **Permanently Archive** ![](images/icon-delete.png) to archive a model package, which also removes it from the **Model Packages** list.
---
title: Custom model assembly
description: Describes how to assemble custom models and environments.

---

# Custom model assembly {: #custom-model-assembly }

While DataRobot provides hundreds of built-in models, there are situations where you need preprocessing or modeling methods that are not currently supported out of the box. To create a custom inference model, you must provide a serialized model artifact with a file extension corresponding to the chosen environment language and any additional custom code required to use the model.

Before adding custom models and environments to DataRobot, you must prepare and structure the files required to run them successfully. The tools and templates necessary to prepare custom models are hosted in the <a target="_blank" href="https://github.com/datarobot/datarobot-user-models">Custom Model GitHub repository</a>. ({% include 'includes/github-sign-in.md' %}) DataRobot recommends understanding the following requirements to prepare your custom model for upload to the Custom Model Workshop.

Topic | Describes
------|-----------
[Custom model components](custom-model-components) | How to identify the components required to run custom inference models.
[Assemble structured custom models](structured-custom-models) | How to assemble and validate structured custom models compatible with DataRobot.
[Assemble unstructured custom models](unstructured-custom-models) | How to assemble and validate unstructured custom models compatible with DataRobot.
[DRUM CLI tool](custom-model-drum) | How to download and install the DataRobot user model (DRUM) to work with and test custom models and custom environments locally before uploading to DataRobot.
[Test a custom model locally](custom-local-test) | How to test custom inference models in your local environment using the DataRobot Model Runner (DRUM) tool.
[GitHub Actions for custom models](custom-model-github-action) | The custom models action manages custom inference models and deployments in DataRobot via GitHub CI/CD workflows.
---
title: Unlock Holdout
description: Holdout, the portion of your data DataRobot reserves when building models, provides an evaluation metric that measures a model's accuracy against the unseen data to validate model quality.

---


# Unlock Holdout {: #unlock-holdout }

The *Holdout* column displays an evaluation metric that measures a model's accuracy against unseen ("new") data. Holdout is calculated using the trained model's predictions on the [holdout partition](data-partitioning). DataRobot reserves a portion of your data to use as holdout (20% by default); it does not train models using this data but instead validates the quality of your models once they have been trained.

!!! tip
    You should only unlock your holdout data after having made all your model-related decisions. **Once your project's holdout has been unlocked, it cannot be re-locked.**

!!! note
        If you run full or Quick Autopilot and DataRobot returns a model [recommended and prepared for deployment](model-rec-process), the specifics described below work slightly differently.

To display a specific model's Holdout score:

1.  Click **Unlock project Holdout for all models** on the rightmost panel.

    ![](images/holdout-unlock.png)

2.  Confirm your decision by clicking **Unlock holdout**.


When you unlock holdout, the label on the project menu changes to **Holdout is unlocked** and a value displays in the Holdout column.

![](images/holdout-unlock-after.png)

Once you have unlocked the holdout data, view the Leaderboard scores on the test data. Then, look at the [Lift Chart](lift-chart). Alternate the **Data Source** dropdown between Validation and Holdout to determine the accuracy of the model's predictions.
---
title: V9.0.3
description: DataRobot Release 9.0.3 release notes.

---

# V9.0.3 {: #v903 }

_June 30, 2023_

The DataRobot v9.0.3 release includes some improvements and fixed issues in the DataRobot Self-Managed AI Platform platform. See the v9.0.0 release notes for:

* [Features introduced in v9.0.0](v9.0/index)

Version 9.0.3 now supports Azure AKS v1.25 in addition to OpenShift 4.10 and AWS EKS with K8s v1.23.

## Issues fixed in v9.0.3 {: #issues-fixed-in-v903 }

The following issues have been fixed since release v9.0.3:

### MLOps {: #mlops }

- RAPTOR-9471: Fixes an issue making it impossible for custom models with training data to be shared with any other role except `consumer`.

### Predictions {: #predictions }

- PRED-8839: Fixes an issue causing Portable Prediction Servers to crash when requesting Prediction Explanations for models trained on Autopilot and cross-validation finished.


_All product and company names are trademarks&trade; or registered&reg; trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them_.
---
title: Demand forecasting and retraining workflow
description: Implement retraining policies with DataRobot MLOps demand forecast deployments.

---

# Demand forecasting and retraining workflow {: #demand-forecasting-and-retraining-workflow }

[Access this AI accelerator on GitHub <span style="vertical-align: sub">:material-arrow-right-circle:{.lg }</span>](https://github.com/datarobot-community/ai-accelerators/blob/main/end-to-end/Demand_forecasting_retraining/End_to_end_demand_forecasting_retraining.ipynb){ .md-button }

This accelerator  demonstrates retraining policies with DataRobot MLOps demand forecast deployments.

This accelerator is a another installment of a series on demand forecasting. The [first accelerator](demand-flow) focuses on handling common data and modeling challenges, identifies common pitfalls in real-life time series data, and provides helper functions to scale experimentation. The [second accelerator](cold-start) provides the building blocks for cold start modeling workflow on series with limited or no history. They can be used as a starting point to create a model deployment for the app. The [third accelerator](ml-what-if) is a what-if app that allows you to adjust certain known in advance variable values to see how changes in those factors might affect the forecasted demand.
# Treasure Data {: #treasure-data }

## Supported authentication {: #supported-authentication }

- Username/password

## Prerequisites {: #prerequisites }

The following is required before connecting to Treasure Data (TD-Hive) in DataRobot:

- Treasure Data account

## Required parameters {: #required-parameters }

The table below lists the minimum required fields to establish a connection with Treasure Data:

Required field | Description |  Documentation
--------------- | ---------- |  -----------
`address`       | The connection URL that supplies connection information for Treasure Data.  |   [Treasure Data documentation](https://docs.treasuredata.com/display/public/PD/JDBC+Driver+for+Hive+Query+Engine)
`database`  | A unique identifier for your database.  |  [Treasure Data documentation](https://docs.treasuredata.com/display/public/PD/Database+and+Table+Management)

Learn about additional [configuration options for Treasure Data](https://docs.treasuredata.com/display/public/PD/JDBC+Driver+for+Hive+Query+Engine).

{% include 'includes/data-conn-trouble.md' %}
---
title: Make predictions with the API
description: DataRobot's Prediction API provides a mechanism for using your model for real-time predictions on a prediction server.

---


# Make predictions with the API {: #make-predictions-with-the-api }

DataRobot's Prediction API provides a mechanism for using your model for real-time predictions on an external application server. Follow [the guidelines](dr-predapi) for making predictions with the Prediction API. You can also review how to [retrieve a prediction server ID](pred-server-id) using cURL commands from the REST API or by using the DataRobot Python client to make predictions with a deployment.
---
title: Add data
description: In Workbench, you can add datasets from a local file, data connection, or the Data Registry.

---

# Add data {: #add-data }

From anywhere in a Use Case, you can add data by clicking **Add new > Add datasets**, opening the **Add data** modal. By adding data before setting up an experiment, you have the chance to [explore the dataset](wb-data-tab#view-exploratory-data-insights) and decide if it's ready for modeling.

![](images/wb-add-data-1.png)

This section covers the following topics:

Topic     | Describes...
---------- | -----------
[Local file](wb-local-file)      | Upload datasets stored locally on your computer.
[Data connection](wb-connect)     | Connect to and add data from an external data source.
[Data Registry](wb-data-registry)      | Add any static or snapshot datasets you currently have access to in the AI Catalog.
---
title: Understand tabs
description: The Understand tabs—Feature Effects, Feature Impact, Prediction Explanations, and Word Cloud—explain what drives a model’s predictions.

---

# Understand {: #understand }

The **Understand** tabs explain what drives a model’s predictions:

Leaderboard tabs  | Description | Source
------------------|-------------|------------
[Feature Impact](feature-impact)  | Provides a high-level visualization that identifies which features are most strongly driving model decisions. | Training data
[Feature Effects](feature-effects) | Visualizes the effect of changes in the value of each feature on the model’s predictions. | Training data prior to v5.0; Training, Validation, Holdout (selectable) in v5.0+
[Cluster Insights](cluster-insights) | Visualizes the groupings of data that result from modeling in [clustering](clustering) mode, an [unsupervised learning](glossary/index#unsupervised-learning) technique. | Training data
[Prediction Explanations](pred-explain/index) | Illustrates what drives predictions on a row-by-row basis using XEMP or SHAP methodology. | Low and high thresholds and baseline prediction based on Validation data (XEMP) or training data (SHAP)
[Word Cloud](analyze-insights#word-cloud-insights)  | Displays the most relevant words and short phrases in word cloud format. | Training data
---
title: Normalizing for monotonicity
description: With DataRobot's monotonicity, to normalize or not to normalize, that is the question.
---

# Normalizing for monotonicity {: #normalizing-for-monotonicity }

<span style="color:red;font-size: 1rem"> `Robot 1`</span>

**To normalize or not to normalize, that is the question**

Got some questions about monotonicity (I've been testing models heavily with such constraints). Would appreciate any answers or documentation that can help.

1. When we apply monotonic Increasing/Decreasing constraints to attributes, is DataRobot doing some kind of normalization (capping amd flooring, binning etc.)? 
2. When we apply `try only monotonic` models, will it try GBM, XGBOOST, RF etc.?

Thanks for the help!

<span style="color:red;font-size: 1rem"> `Robot 2`</span>

No normalization that I know of and just xgboost and gam. You don't really need to normalize data for xgboost models.

<span style="color:red;font-size: 1rem"> `Robot 3`</span>

For docs you can check out how to [configure feature constraints](feature-con#monotonicty) and then the [workflow to build them](monotonic).  And here's an [ELI5](eli5) answer to "what does monotonic mean?"
---
title: Document AI overview
description: Read background information and a simplified workflow overview.
section_name: AutoML
maturity: public-preview
---


# Document AI overview {: #document-ai-overview }

Analysts and data scientists often want to use the information contained in PDF documents to build models. However, manually intensive data preparation requirements present a challenging barrier to efficient use of documents as a data source. Often the volume of documents is large enough that reading through each or manually formatting and preparing them into tabular formats is not feasible. Information spread out in a large corpus of documents&mdash;in a variety of formats with inconsistencies&mdash;makes the frequently valuable text information contained within these documents inaccessible. 

Document AI provides a way to build models on raw PDF documents without manually intensive data preparation steps. It provides end-to-end support for PDFs with encoded text that is readily machine readable:

* DocumentTextExtractor (DTE): Extracts embedded text from a PDF document. Example: Save a document written on your computer as PDF, then upload it. 

* Optical Character Recognition (OCR): Extracts scanned text. Example: You print out a document and then scan it and upload it as PDF. Content is seen as pixels (not as “known” text).

Document AI works with many project types, including regression, binary and multiclass classification, multilabel, clustering, and anomaly detection. The process extracts content and categorizes it as type `document` for modeling:

![](images/docai-1.png)

Projects can include not only one or more `document` features, but any other feature type that DataRobot supports.


## Workflow overview {: #workflow-overview }

Following is the Document AI workflow:

1. Create a [PDF-based dataset](doc-ai-ingest) for use in projects via the AI Catalog or local file upload.

2. Preview documents for potential [data quality](doc-ai-ingest#data-quality) issues.

3. Build models using the standard DataRobot workflow.

4. Evaluate models on the Leaderboard with [document-specific insights](doc-ai-insights).

7. Select a model to use for [making predictions](doc-ai-predictions) via Make Predictions, the DataRobot API, or batch predictions.

## Feature considerations {: #feature-considerations }

- Time series projects are not supported.