Package: airflow-provider-datarobot
DataRobot Provider for Apache AirflowThis package provides operators, sensors, and a hook to integrateDataRobotinto Apache Airflow.
Using these components, you should be able to build the essential DataRobot pipeline - create a project, train models, deploy a model,
and score predictions against the model deployment.Install the Airflow providerThe DataRobot provider for Apache Airflow requires an environment with the following dependencies installed:Apache Airflow>= 2.3DataRobot Python API Client>= 3.2.0To install the DataRobot provider, run the following command:pipinstallairflow-provider-datarobotCreate a connection from Airflow to DataRobotThe next step is to create a connection from Airflow to DataRobot:In the Airflow user interface, clickAdmin > Connectionstoadd an Airflow connection.On theList Connectionpage, click+ Add a new record.In theAdd Connectiondialog box, configure the following fields:FieldDescriptionConnection Iddatarobot_default(this name is used by default in all operators)Connection TypeDataRobotAPI KeyA DataRobot API key, created in theDataRobot Developer Tools, from theAPI Keyssection.DataRobot endpoint URLhttps://app.datarobot.com/api/v2by defaultClickTestto establish a test connection between Airflow and DataRobot.When the connection test is successful, clickSave.Create preconfigured connections to DataRobotYou can create preconfigured connections to store and manage credentials to use with Airflow Operators,
replicating theconnection on the DataRobot side.Currently, the supported credential types are:CredentialsDescriptionDataRobot Basic CredentialsLogin/password pairsDataRobot GCP CredentialsGoogle Cloud Service account keyDataRobot AWS CredentialsAWS access keysDataRobot Azure Storage CredentialsAzure Storage secretDataRobot OAuth CredentialsOAuth tokensDataRobot JDBC DataSourceJDBC connection attributesAftercreating a preconfigured connection through the Airflow UI or API,
you can access your stored credentials withGetOrCreateCredentialOperatororGetOrCreateDataStoreOperatorto replicate them in DataRobot and retrieve the correspondingcredentials_idordatastore_id.JSON configuration for the DAG runOperators and sensors use parameters from theconfigJSON submitted when triggering the DAG; for example:{"training_data":"s3-presigned-url-or-local-path-to-training-data","project_name":"ProjectcreatedfromAirflow","autopilot_settings":{"target":"readmitted"},"deployment_label":"DeploymentcreatedfromAirflow","score_settings":{"intake_settings":{"type":"s3","url":"s3://path/to/scoring-data/Diabetes10k.csv","credential_id":"<credential_id>"},"output_settings":{"type":"s3","url":"s3://path/to/results-dir/Diabetes10k_predictions.csv","credential_id":"<credential_id>"}}}These config values are accessible in theexecute()method of any operator in the DAG
through thecontext["params"]variable; for example, to get training data, you could use the following:defexecute(self,context:Dict[str,Any])->str:...training_data=context["params"]["training_data"]...ModulesOperatorsGetOrCreateCredentialOperatorFetches a credential by name. This operator attempts to find a DataRobot credential with the provided name. If the credential doesn't exist, the operator creates it using the Airflow preconfigured connection with the same connection name.Returns a credential ID.Required config parameters:ParameterTypeDescriptioncredentials_param_namestrThe name of parameter in the config file for the credential name.GetOrCreateDataStoreOperatorFetches a DataStore by Connection name. If the DataStore does not exist, the operator attempts to create it using Airflow preconfigured connection with the same connection name.Returns a credential ID.Required config params:ParameterTypeDescriptionconnection_param_namestrThe name of the parameter in the config file for the connection name.CreateDatasetFromDataStoreOperatorLoads a dataset from a JDBC Connection to the DataRobot AI Catalog.Returns a dataset ID.Required config params:ParameterTypeDescriptiondatarobot_jdbc_connectionstrThe existing preconfigured DataRobot JDBC connection name.dataset_namestrThe name of the loaded dataset.table_schemastrThe database table schema.table_namestrThe source table name.do_snapshotboolIfTrue, creates a snapshot dataset. IfFalse, creates a remote dataset. If unset, uses the server default (True). Creating snapshots from non-file sources may be disabled by theDisable AI Catalog Snapshotspermission.persist_data_after_ingestionboolIfTrue, enforce saving all data (for download and sampling) and allow a user to view the extended data profile (which includes data statistics like min, max, median, mean, histogram, etc.). IfFalse, don't enforce saving data. If unset, uses the server default (True). The data schema (feature names and types) will still be available. Specifying this parameter toFalseanddoSnapshottoTrueresults in an error.UploadDatasetOperatorUploads a local file to the DataRobot AI Catalog.Returns a dataset ID.Required config params:ParameterTypeDescriptiondataset_file_pathstrThe local path to the training dataset.UpdateDatasetFromFileOperatorCreates a new dataset version from a file.Returns a dataset version ID when the new version uploads successfully.Required config params:ParameterTypeDescriptiondataset_idstrThe DataRobot AI Catalog dataset ID.dataset_file_pathstrThe local path to the training dataset.CreateDatasetVersionOperatorCreates a new version of the existing dataset in the AI Catalog.Returns a dataset version ID.Required config params:ParameterTypeDescriptiondataset_idstrThe DataRobot AI Catalog dataset ID.datasource_idstrThe existing DataRobot datasource ID.credential_idstrThe existing DataRobot credential ID.CreateOrUpdateDataSourceOperatorCreates a data source or updates it if it already exists.Returns a DataRobot DataSource ID.Required config params:ParameterTypeDescriptiondata_store_idstrTHe DataRobot datastore ID.CreateProjectOperatorCreates a DataRobot project.Returns a project ID.Several options of source dataset supported:Local file or pre-signed S3 URLCreate a project directly from a local file or a pre-signed S3 URL.Required config params:ParameterTypeDescriptiontraining_datastrThe pre-signed S3 URL or the local path to the training dataset.project_namestrThe project name.Note:In case of an S3 input, thetraining_datavalue must be apre-signed AWS S3 URL.AI Catalog dataset from config fileCreate a project from an existing dataset in the DataRobot AI Catalog using a dataset ID defined in the config file.Required config params:ParameterTypeDescriptiontraining_dataset_idstrThe dataset ID corresponding to existing dataset in the DataRobot AI Catalog.project_namestrThe project name.AI Catalog dataset from previous operatorCreate a project from an existing dataset in the DataRobot AI Catalog using a dataset ID from the previous operator.
In this case, your previous operator must return a valid dataset ID (for exampleUploadDatasetOperator)
and you should use this output value as adataset_idargument in theCreateProjectOperatorobject creation step.Required config params:ParameterTypeDescriptionproject_namestrThe project name.For moreproject settings, see the DataRobot documentation.TrainModelsOperatorRuns DataRobot Autopilot to train models.ReturnsNone.Parameters:ParameterTypeDescriptionproject_idstrThe DataRobot project ID.Required config params:ParameterTypeDescriptiontargetstrThe name of the column defining the modeling target."autopilot_settings":{"target":"readmitted"}For moreautopilot settings, see the DataRobot documentation.DeployModelOperatorDeploy a specified model.Returns a deployment ID.Parameters:ParameterTypeDescriptionmodel_idstrThe DataRobot model ID.Required config params:ParameterTypeDescriptiondeployment_labelstrThe deployment label name.For moredeployment settings, see the DataRobot documentation.DeployRecommendedModelOperatorDeploys a recommended model.Returns a deployment ID.Parameters:ParameterTypeDescriptionproject_idstrThe DataRobot project ID.Required config params:ParameterTypeDescriptiondeployment_labelstrThe deployment label name.For moredeployment settings, see the DataRobot documentation.ScorePredictionsOperatorScores batch predictions against the deployment.Returns a batch prediction job ID.Prerequisites:UseGetOrCreateCredentialOperatorto pass acredential_idfrom the preconfigured DataRobot Credentials (Airflow Connections) or manually set thecredential_idparameter in the config.Note:You canadd S3 credentials to DataRobot via the Python API client.Oruse a Dataset ID from the DataRobot AI Catalog.Oruse a DataStore ID for a JDBC source connection; you can useGetOrCreateDataStoreOperatorto passdatastore_idfrom a preconfigured Airflow Connection.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.intake_datastore_idstrThe DataRobot datastore ID for the JDBC source connection.output_datastore_idstrThe DataRobot datastore ID for the JDBC destination connection.intake_credential_idstrThe DataRobot credentials ID for the source connection.output_credential_idstrThe DataRobot credentials ID for the destination connection.Sample config: Pre-signed S3 URL"score_settings":{"intake_settings":{"type":"s3","url":"s3://my-bucket/Diabetes10k.csv",},"output_settings":{"type":"s3","url":"s3://my-bucket/Diabetes10k_predictions.csv",}}Sample config: Pre-signed S3 URL with a manually set credential ID"score_settings":{"intake_settings":{"type":"s3","url":"s3://my-bucket/Diabetes10k.csv","credential_id":"<credential_id>"},"output_settings":{"type":"s3","url":"s3://my-bucket/Diabetes10k_predictions.csv","credential_id":"<credential_id>"}}Sample config: Scoring dataset in the AI Catalog"score_settings":{"intake_settings":{"type":"dataset","dataset_id":"<datasetId>",},"output_settings":{}}For morebatch prediction settings, see the DataRobot documentation.GetTargetDriftOperatorGets the target drift from a deployment.Returns a dict with the target drift data.Parameters:ParameterTypeDescriptiondeployment_idstrTHe DataRobot deployment ID.No config params are required; however, theoptional paramsmay be passed in the config as follows:"target_drift":{}GetFeatureDriftOperatorGets the feature drift from a deployment.Returns a dict with the feature drift data.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.No config params are required; however, theoptional paramsmay be passed in the config as follows:"feature_drift":{}GetServiceStatsOperatorGets service stats measurements from a deployment.Returns a dict with the service stats measurements data.Parameters:ParameterTypeDescriptiondeployment_idstrTHe DataRobot deployment ID.No config params are required; however, theoptional paramsmay be passed in the config as follows:"service_stats":{}GetAccuracyOperatorGets the accuracy of a deployment’s predictions.Returns a dict with the accuracy for a deployment.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.No config params are required; however, theoptional paramsmay be passed in the config as follows:"accuracy":{}GetBiasAndFairnessSettingsOperatorGets the Bias And Fairness settings for deployment.Returns a dict with theBias And Fairness settings for a Deployment.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.No config params are required.UpdateBiasAndFairnessSettingsOperatorUpdates the Bias And Fairness settings for deployment.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.Sample config params:"protected_features": ["attribute1"],
"preferable_target_value": "True",
"fairness_metrics_set": "equalParity",
"fairness_threshold": 0.1,GetSegmentAnalysisSettingsOperatorGets the segment analysis settings for a deployment.Returns a dict with thesegment analysis settings for a deploymentParameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.No config params are required.UpdateSegmentAnalysisSettingsOperatorUpdates the segment analysis settings for a deployment.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.Sample config params:"segment_analysis_enabled": True,
"segment_analysis_attributes": ["attribute1", "attribute2"],GetMonitoringSettingsOperatorGets the monitoring settings for deployment.Returns a dict with the config params for a deployment.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.No config params are required.Sample monitoring settings:{
"drift_tracking_settings": {  } 
"association_id_settings": {  }
"predictions_data_collection_settings": {  }
}DictionaryDescriptiondrift_tracking_settingsThedrift tracking settings for this deployment.association_id_settingsTheassociation ID settings for this deployment.predictions_data_collection_settingsThepredictions data collection settings of this deployment.UpdateMonitoringSettingsOperatorUpdates monitoring settings for a deployment.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.Sample config params:"target_drift_enabled": True,
"feature_drift_enabled": True,
"association_id_column": ["id"],
"required_association_id": False,
"predictions_data_collection_enabled": False,BatchMonitoringOperatorCreates a batch monitoring job for the deployment.Returns a batch monitoring job ID.Prerequisites:UseGetOrCreateCredentialOperatorto pass acredential_idfrom the preconfigured DataRobot Credentials (Airflow Connections) or manually set thecredential_idparameter in the config.Note:You canadd S3 credentials to DataRobot via the Python API client.Oruse a Dataset ID from the DataRobot AI Catalog.Oruse a DataStore ID for a JDBC source connection; you can useGetOrCreateDataStoreOperatorto passdatastore_idfrom a preconfigured Airflow Connection.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.datastore_idstrThe DataRobot datastore ID.credential_idstrThe DataRobot credentials ID.Sample config params:Sample config"deployment_id":"61150a2fadb5586af4118980","monitoring_settings":{"intake_settings":{"type":"bigquery","dataset":"integration_example_demo","table":"actuals_demo","bucket":"datarobot_demo_airflow",},"monitoring_columns":{"predictions_columns":[{"class_name":"True","column_name":"target_True_PREDICTION"},{"class_name":"False","column_name":"target_False_PREDICTION"},],"association_id_column":"id","actuals_value_column":"ACTUAL",},}Sample config: Manually set credential ID"deployment_id":"61150a2fadb5586af4118980","monitoring_settings":{"intake_settings":{"type":"bigquery","dataset":"integration_example_demo","table":"actuals_demo","bucket":"datarobot_demo_airflow","credential_id":"<credential_id>"},"monitoring_columns":{"predictions_columns":[{"class_name":"True","column_name":"target_True_PREDICTION"},{"class_name":"False","column_name":"target_False_PREDICTION"},],"association_id_column":"id","actuals_value_column":"ACTUAL",},}For morebatch monitoring settings, see the DataRobot documentation.DownloadModelScoringCodeOperatorDownloads scoring code artifact from a model.Parameters:ParameterTypeDescriptionproject_idstrThe DataRobot project ID.model_idstrThe DataRobot model ID.base_pathstrThe base path for storing a downloaded model artifact.Sample config params:"source_code": False,For morescoring code download parameters, see the DataRobot documentation.DownloadDeploymentScoringCodeOperatorDownloads scoring code artifact from a deployment.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.base_pathstrThe base path for storing a downloaded model artifact.Sample config params:"source_code": False,
"include_agent": False,
"include_prediction_explanations": False,
"include_prediction_intervals": False,For morescoring code download parameters, see the DataRobot documentation.SubmitActualsFromCatalogOperatorDownloads scoring code artifact from a deployment.Returns an actuals upload job ID.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.dataset_idstrThe DataRobot AI Catalog dataset ID.dataset_version_idstrThe DataRobot AI Catalog dataset version ID.Sample config params:"association_id_column": "id",
"actual_value_column": "ACTUAL",
"timestamp_column": "timestamp",StartAutopilotOperatorTriggers DataRobot Autopilot to train a set of models.Parameters:ParameterTypeDescriptionproject_idstrThe DataRobot project ID.featurelist_idstrSpecifies which feature list to use.relationships_configuration_idstrID of the relationships configuration to use.segmentation_task_idstrID of the segementation task to use.Sample config params:"autopilot_settings": {
    "target": "column_name",
    "mode": AUTOPILOT_MODE.QUICK,
}For moreanalyze_and_model parameters, see the DataRobot documentation.CreateExecutionEnvironmentOperatorCreate an execution environment.Returns an execution environment ID.Parameters:ParameterTypeDescriptionnamestrThe execution environment name.descriptionstrThe execution environment description.programming_languagestrThe programming language of the environment to be created.Sample config params:"execution_environment_name": "My Demo Env",
"custom_model_description": "This is a custom model created by Airflow",
"programming_language": "python",For moreexecution environment creation parameters, see the DataRobot documentation.CreateExecutionEnvironmentVersionOperatorCreate an execution environment version.Returns a version ID for the newly created execution environment .Parameters:ParameterTypeDescriptionexecution_environment_idstrThe ID of the execution environment.docker_context_pathstrThe file path to a Docker context archive or folder.environment_version_labelstrA short, human-readable string to label the environment version.environment_version_descriptionstrThe execution environment version description.For moreexecution environment version creation parameters, see the DataRobot documentation.CreateCustomInferenceModelOperatorCreate a custom inference model.Returns the ID for the created custom model.Parameters:ParameterTypeDescriptionnamestrName of the custom model.descriptionstrDescription of the custom model.Sample DAG config params:"target_type": - Target type of the custom inference model.
    Values: [`datarobot.TARGET_TYPE.BINARY`, `datarobot.TARGET_TYPE.REGRESSION`,
    `datarobot.TARGET_TYPE.MULTICLASS`, `datarobot.TARGET_TYPE.UNSTRUCTURED`]
"target_name": - Target feature name.
    It is optional (ignored if provided) for `datarobot.TARGET_TYPE.UNSTRUCTURED` target type.
"programming_language": - Programming language of the custom learning model.
"positive_class_label": - Custom inference model positive class label for binary classification.
"negative_class_label": - Custom inference model negative class label for binary classification.
"prediction_threshold": - Custom inference model prediction threshold.
"class_labels": - Custom inference model class labels for multiclass classification.
"network_egress_policy": - Determines whether the given custom model is isolated, or can access the public network.
"maximum_memory": - The maximum memory that might be allocated by the custom model.
"replicas": - A fixed number of replicas that will be deployed in the cluster.For morecustom inference model creation parameters, see the DataRobot documentation.CreateCustomModelVersionOperatorCreate a custom model version.Returns the version ID for the created custom model.Parameters:ParameterTypeDescriptioncustom_model_idstrThe ID of the custom model.base_environment_idstrThe ID of the base environment to use with the custom model version.training_dataset_idstrThe ID of the training dataset to assign to the custom model.holdout_dataset_idstrThe ID of the holdout dataset to assign to the custom model.custom_model_folderstrThe path to a folder containing files to be uploaded. Each file in the folder is uploaded under path relative to a folder path.create_from_previousboolIf set to True, this parameter creates a custom model version containing files from a previous version.Sample DAG config params:"is_major_update" - The flag defining if a custom model version will be a minor or a major version.
"files" - The list of tuples, where values in each tuple are the local filesystem path and
            the path the file should be placed in the model.
"files_to_delete" - The list of a file items IDs to be deleted.
"network_egress_policy": - Determines whether the given custom model is isolated, or can access the public network.
"maximum_memory": - The maximum memory that might be allocated by the custom model.
"replicas": - A fixed number of replicas that will be deployed in the cluster.
"required_metadata_values" - Additional parameters required by the execution environment.
"keep_training_holdout_data" - If the version should inherit training and holdout data from the previous version.For morecustom inference model creation parameters, see the DataRobot documentation.CustomModelTestOperatorCreate and start a custom model test.Returns an ID for the custom model test.Parameters:ParameterTypeDescriptioncustom_model_idstrThe ID of the custom model.custom_model_version_idstrThe ID of the custom model version.dataset_idstrThe ID of the testing dataset forstructured custom models. Ignored and not required forunstructured models.Sample DAG config params:"network_egress_policy": - Determines whether the given custom model is isolated, or can access the public network.
"maximum_memory": - The maximum memory that might be allocated by the custom model.
"replicas": - A fixed number of replicas that will be deployed in the cluster.For morecustom model test creation parameters, see the DataRobot documentation.GetCustomModelTestOverallStatusOperatorGet the overall status for custom model tests.Returns the custom model test status.Parameters:ParameterTypeDescriptioncustom_model_test_idstrThe ID of the custom model test.For morecustom model test get status parameters, see the DataRobot documentation.CreateCustomModelDeploymentOperatorCreate a deployment from a DataRobot custom model image.Returns the deployment ID.Parameters:ParameterTypeDescriptioncustom_model_version_idstrThe ID of the deployed custom model.deployment_namestrA human-readable label for the deployment.default_prediction_server_idstrAn identifier for the default prediction server.descriptionstrA human-readable description of the deployment.importancestrThe deployment importance level.For morecreate_from_custom_model_version parameters, see the DataRobot documentation.GetDeploymentModelOperatorGets information about the deployment's current model.Returns a model information from a deployment.Parameters:ParameterTypeDescriptiondeployment_idstrAn identifier for the deployed model.For moreget deployment parameters, see the DataRobot documentation.ReplaceModelOperatorReplaces the current model for a deployment.Returns model information for the mode replacing the deployed model.Parameters:ParameterTypeDescriptiondeployment_idstrAn identifier for the deployed model.new_model_idstrThe ID of the replacement model. If you are replacing the deployment's model with a custom inference model, you must use a specific custom model version ID.reasonstrThe reason for the model replacement. Must be one of 'ACCURACY', 'DATA_DRIFT', 'ERRORS', 'SCHEDULED_REFRESH', 'SCORING_SPEED', or 'OTHER'. This value will be stored in the model history to keep track of why a model was replaced.For morereplace_model parameters, see the DataRobot documentation.ActivateDeploymentOperatorActivate or deactivate a Deployment.Returns the Deployment status (active or inactive).Parameters:ParameterTypeDescriptiondeployment_idstrAn identifier for the deployed model.activatestrIf set to True, this parameter activates the deployment. Set to False to deactivate the deployment.For moreactivate deployment parameters, see the DataRobot documentation.GetDeploymentStatusOperatorGet the deployment status (active or inactive).Returns the deployment status.Parameters:ParameterTypeDescriptiondeployment_idstrAn identifier for the deployed model.For moredeployment parameters, see the DataRobot documentation.RelationshipsConfigurationOperatorCreates a relationship configuration.Returns the relationships configuration ID.Parameters:ParameterTypeDescriptiondataset_definitionsstrA list of dataset definitions. Each element is a dict retrieved from theDatasetDefinitionOperatoroperator.relationshipsstrA list of relationships. Each element is a dict retrieved from DatasetRelationshipOperator operator.feature_discovery_settingsstrOptional. A list of Feature Discovery settings. If not provided, it will be retrieved from the DAG configuration parameters. Otherwise, default settings are used.For moreFeature Discovery parameters, see the DataRobot documentation.DatasetDefinitionOperatorCreates a dataset definition for Feature Discovery.Returns a dataset definition dict.Parameters:ParameterTypeDescriptiondataset_identifierstrThe alias of the dataset, used directly as part of the generated feature names.dataset_idstrThe identifier of the dataset in the AI Catalog.dataset_version_idstrThe identifier of the dataset version in the AI Catalog.primary_temporal_keystrThe name of the column indicating the time of record creation.feature_list_idstrSpecifies the feature list to use.snapshot_policystrThe policy to use when creating a project or making predictions. If omitted, the endpoint will use 'latest' by default.For morecreate-dataset-definitions-and-relationships-using-helper-functions, see the DataRobot documentation.DatasetRelationshipOperatorCreate a relationship between datasets defined in DatasetDefinition.Returns a dataset definition dict.Parameters:ParameterTypeDescriptiondataset1_identifierList[str]Identifier of the first dataset in this relationship. This is specified in the identifier field of thedataset_definitionstructure. If set to None, then the relationship is with the primary dataset.dataset2_identifierList[str]Identifier of the second dataset in this relationship. This is specified in the identifier field of thedataset_definitionschema.dataset1_keysList[str]A list of strings (max length: 10 min length: 1). The column(s) from the first dataset which are used to join to the second dataset.dataset2_keysList[str]A list of strings (max length: 10 min length: 1). The column(s) from the second dataset that are used to join to the first dataset.feature_derivation_window_startintHow many time units of each dataset's primary temporal key into the past relative to the datetimePartitionColumn the feature derivation window should begin. Will be a negative integer, If present, the feature engineering graph performs time-aware joins.feature_derivation_window_endintDetermines how many units of time of each dataset's record primary temporal key into the past relative to the datetimePartitionColumn the feature derivation window should end.  It is a non-positive integer if present. If present, the feature engineering graph performs time-aware joins.feature_derivation_window_time_unitstrThe unit of time the feature derivation window. One ofdatarobot.enums.AllowedTimeUnitsSAFERIf present, time-aware joins will be used. Only applicable when dataset1_identifier is not provided.feature_derivation_windowsListA list of feature derivation windows settings. If present, time-aware joins will be used. Only allowed whenfeature_derivation_window_start,feature_derivation_window_end, andfeature_derivation_window_time_unitare not provided.prediction_point_roundingList[dict]Closest value ofprediction_point_rounding_time_unitto round the prediction point into the past when applying the feature derivation if present. Only applicable whendataset1_identifieris not provided.prediction_point_rounding_time_unitstrTime unit of the prediction point rounding. One ofdatarobot.enums.AllowedTimeUnitsSAFER. Only applicable whendataset1_identifieris not provided.For morecreate-dataset-definitions-and-relationships-using-helper-functions, see the DataRobot documentation.ComputeFeatureImpactOperatorCreates a Feature Impact job in DataRobot.Returns a Feature Impact job ID.Parameters:ParameterTypeDescriptionproject_idstrDataRobot project ID.model_idstrDataRobot model ID.For morerequest_feature_impact, see the DataRobot documentation.ComputeFeatureEffectsOperatorSubmit a request to compute Feature Effects for the model.Returns the Feature Effects job ID.Parameters:ParameterTypeDescriptionproject_idstrDataRobot project ID.model_idstrDataRobot model ID.For morerequest_feature_impact parameters, see the DataRobot documentation.ComputeShapOperatorSubmit a request to compute a SHAP impact job for the model.Returns a SHAP impact job ID.Parameters:ParameterTypeDescriptionproject_idstrDataRobot project ID.model_idstrDataRobot model ID.For moreshap-impact parameters, see the DataRobot documentation.CreateExternalModelPackageOperatorCreate an external model package in DataRobot MLOps from JSON configuration.Returns a model package ID of the newly created model package.Parameters:ParameterTypeDescriptionmodel_infostrA JSON object of external model parameters.Example of JSON configuration for a regression model:.. code-block:: python{
     "name": "Lending club regression",
     "modelDescription": {
             "description": "Regression on lending club dataset"
         }
     "target": {
         "type": "Regression",
         "name": "loan_amnt"
     }
}Example JSON for a binary classification model:.. code-block:: python{
    "name": "Surgical Model",
    "modelDescription": {
        "description": "Binary classification on surgical dataset",
        "location": "/tmp/myModel"
        },
        "target": {
            "type": "Binary",
            "name": "complication",
            "classNames": ["Yes","No"],  # minority/positive class should be listed first
            "predictionThreshold": 0.5
        }
    }
}Example JSON for a multiclass classification model:.. code-block:: python{
    "name": "Iris classifier",
    "modelDescription": {
    "description": "Classification on iris dataset",
    "location": "/tmp/myModel"
},
    "target": {
        "type": "Multiclass",
        "name": "Species",
        "classNames": [
            "Iris-versicolor",
            "Iris-virginica",
            "Iris-setosa"
        ]
    }
}DeployModelPackageOperatorCreate a deployment from a DataRobot model package.Returns the created deployment ID.Parameters:ParameterTypeDescriptiondeployment_namestrA human readable label of the deployment.model_package_idstrThe ID of the DataRobot model package to deploy.default_prediction_server_idstrAn identifier of a prediction server to be used as the default prediction server. When working with prediction environments, the default prediction server ID should not be provided.prediction_environment_idstrAn identifier of a prediction environment to be used for model deployment.descriptionstrA human readable description of the deployment.importancestrDeployment importance level.user_provided_idstrA user-provided unique ID associated with a deployment definition in a remote git repository.additional_metadataDict[str, str]A Key/Value pair dict, with additional metadata.AddExternalDatasetOperatorUpload a new dataset from the AI Catalog to make predictions for a model.Returns an external dataset ID for the model,Parameters:ParameterTypeDescriptionproject_idstrDataRobot project ID.dataset_idstrDataRobot AI Catalog dataset ID.credential_idstrDataRobot credentials ID.dataset_version_idstrDataRobot AI Catalog dataset version ID.For moreupload_dataset_from_catalog parameters, see the DataRobot documentation.RequestModelPredictionsOperatorRequests predictions against a previously uploaded dataset.Returns a model predictions job ID.Parameters:ParameterTypeDescriptionproject_idstrDataRobot project ID.model_idstrDataRobot model ID.external_dataset_idstrDataRobot external dataset ID.For morerequest_predictions, see the DataRobot documentation.TrainModelOperatorSubmit a job to the queue to train a model from a specific blueprint.Returns a model training job ID.Parameters:ParameterTypeDescriptionproject_idstrDataRobot project ID.blueprint_idstrDataRobot blueprint ID.featurelist_idstrThe identifier of the feature list to use. If not defined, the default feature list for this project is used.source_project_idstrThe source project that created theblueprint_id. IfNone, it defaults to looking in this project. Note that you must have read permissions in this project.Example of DAG config params:
{
"sample_pct":
"scoring_type":
"training_row_count":
"n_clusters":
}For morestart-training-a-model, see the DataRobot documentation.RetrainModelOperatorSubmit a job to the queue to retrain a model on a specific sample size and/or custom feature list.Returns a model retraining job ID.Parameters:ParameterTypeDescriptionproject_idstrDataRobot project ID.model_idstrDataRobot model ID.featurelist_idstrThe identifier of the feature list to use. If not defined, the default for this project is used.Example of DAG config params:
{
"sample_pct":
"scoring_type":
"training_row_count":
}For moretrain-a-model-on-a-different-sample-size, see the DataRobot documentation.PredictionExplanationsInitializationOperatorInitialize prediction explanations for a model.Returns a prediction explanations initialization job ID.Parameters:ParameterTypeDescriptionproject_idstrDataRobot project ID.model_idstrDataRobot model ID.For moreprediction-explanations, see the DataRobot documentation.ComputePredictionExplanationsOperatorCreate prediction explanations for the specified dataset.Returns a job ID for the prediction explanations for the specified dataset.Parameters:ParameterTypeDescriptionproject_idstrDataRobot project ID.model_idstrDataRobot model ID.external_dataset_idstrDataRobot external dataset ID.Example of DAG config params:{
"max_explanations"
"threshold_low"
"threshold_high"
}For moreprediction-explanations, see the DataRobot documentation.SensorsAutopilotCompleteSensorChecks if Autopilot is complete.Parameters:ParameterTypeDescriptionproject_idstrThe DataRobot project ID.ScoringCompleteSensorChecks if batch scoring is complete.Parameters:ParameterTypeDescriptionjob_idstrThe batch prediction job ID.MonitoringJobCompleteSensorChecks if a monitoring job is complete.Parameters:ParameterTypeDescriptionjob_idstrThe batch monitoring job ID.BaseAsyncResolutionSensorChecks if the DataRobot Async API call is complete.Parameters:ParameterTypeDescriptionjob_idstrThe DataRobot async API call status check ID.DataRobotJobSensorChecks whether a DataRobot job is complete.Parameters:ParameterTypeDescriptionproject_idstrDataRobot project ID.job_idstrDataRobot job ID.ModelTrainingJobSensorChecks whether a DataRobot model training job is complete.Returns False if the job is not yet completed, and returns PokeReturnValue(True, trained_model.id) if model training has completed.Parameters:ParameterTypeDescriptionproject_idstrDataRobot project ID.job_idstrDataRobot job ID.HooksDataRobotHookA hook to initialize the DataRobot Public API client.PipelineThe modules described above allow you to construct a standard DataRobot pipeline in an Airflow DAG:create_project_op >> train_models_op >> autopilot_complete_sensor >> deploy_model_op >> score_predictions_op >> scoring_complete_sensorExample DAGSSee the thedatarobot_provider/example_dagsdirectory for the example DAGs.You can find the following examples using a preconfigured connection in thedatarobot_provider/example_dagsdirectory:Example DAGDescriptiondatarobot_pipeline_dag.pyRun the basic end-to-end workflow in DataRobot.datarobot_score_dag.pyPerform DataRobot batch scoring.datarobot_jdbc_batch_scoring_dag.pyPerform DataRobot batch scoring with a JDBC data source.datarobot_aws_s3_batch_scoring_dag.pyUse DataRobot AWS Credentials withScorePredictionsOperator.datarobot_gcp_storage_batch_scoring_dag.pyUse DataRobot GCP Credentials withScorePredictionsOperator.datarobot_bigquery_batch_scoring_dag.pyUse DataRobot GCP Credentials withScorePredictionsOperator.datarobot_azure_storage_batch_scoring_dag.pyUse DataRobot Azure Storage Credentials withScorePredictionsOperator.datarobot_jdbc_dataset_dag.pyUpload a dataset to the AI Catalog through a JDBC connection.datarobot_batch_monitoring_job_dag.pyRun a batch monitoring job.datarobot_create_project_from_ai_catalog_dag.pyCreate a DataRobot project from a DataRobot AI Catalog dataset.datarobot_create_project_from_dataset_version_dag.pyCreate a DataRobot project from a specific dataset version in the DataRobot AI Catalog.datarobot_dataset_new_version_dag.pyCreate a new version of an existing dataset in the AI Catalog.datarobot_dataset_upload_dag.pyUpload a local file to the AI Catalog.datarobot_get_datastore_dag.pyCreate a DataRobot data store withGetOrCreateDataStoreOperator.datarobot_jdbc_dataset_dag.pyCreate a DataRobot project from a JDBC data source.datarobot_jdbc_dynamic_dataset_dag.pyCreate a DataRobot project from a JDBC dynamic data source.datarobot_upload_actuals_catalog_dag.pyUpload actuals from the DataRobot AI Catalog.deployment_service_stats_dag.pyGet a deployment's service statistics withGetServiceStatsOperator.deployment_stat_and_accuracy_dag.pyGet a deployment's service statistics and accuracy.deployment_update_monitoring_settings_dag.pyUpdate a deployment's monitoring settings.deployment_update_segment_analysis_settings_dag.pyUpdate a deployment's segment analysis settings.download_scoring_code_from_deployment_dag.pyDownload a Scoring Code JAR file from a DataRobot deployment.advanced_datarobot_pipeline_jdbc_dag.pyRun the advanced end-to-end workflow in DataRobot.datarobot_autopilot_options_pipeline_dag.pyCreates a DataRobot project and starts Autopilot with advanced options.datarobot_custom_model_pipeline_dag.pyCreate an end-to-end workflow with custom models in DataRobot.datarobot_custom_partitioning_pipeline_dag.pyCreate a custom partitioned project and train models.datarobot_datetime_partitioning_pipeline_dag.pyCreate a datetime partitioned project.datarobot_external_model_pipeline_dag.pyAn end-to-end workflow with external models in DataRobot.datarobot_feature_discovery_pipeline_dag.pyCreate a Feature Discovery project and train models.datarobot_timeseries_pipeline_dag.pyCreate a time series DataRobot project.deployment_activate_deactivate_dag.pyAn example of deployment activation/deactivation and getting deployment status.deployment_replace_model_dag.pyAn example of model replacement for deployments.model_compute_insights_dag.pyAn example of computing Feature Impact and Feature Effects.model_compute_prediction_explanations_dag.pyAn example of a compute prediction explanations job.model_compute_predictions_dag.pyAn example of computing predictions for model.model_compute_shap_dag.pyAn example of computing SHAP.model_retrain_dag.pyExample of model retraining job on a specific sample size/featurelist.model_train_dag.pyExample of model training job based on specific blueprint.The advanced end-to-end workflow in DataRobot (advanced_datarobot_pipeline_jdbc_dag.py) contains the following steps:Ingest a dataset to the AI Catalog from JDBC datasourceCreate a DataRobot projectTrain models using AutopilotDeploy the recommended modelChange deployment settings (enable monitoring settings, segment analysis, and bias and fairness)Run batch scoring using a JDBC datasourceUpload actuals from a JDBC datasourceCollect deployment metrics: service statistics, features drift, target drift, accuracy and process it with custom python operator.IssuesPlease submitissuesandpull requestsin our official repo:https://github.com/datarobot/airflow-provider-datarobotWe are happy to hear from you. Please email any feedback to the authors atsupport@datarobot.com.Copyright NoticeCopyright 2023 DataRobot, Inc. and its affiliates.All rights reserved.This is proprietary source code of DataRobot, Inc. and its affiliates.Released under the terms of DataRobot Tool and Utility Agreement.