Package: airflow-provider-datarobot-early-access
DataRobot Provider for Apache AirflowThis package provides operators, sensors, and a hook to integrateDataRobotinto Apache Airflow.
Using these components, you should be able to build the essential DataRobot pipeline - create a project, train models, deploy a model,
and score predictions against the model deployment.Install the Airflow providerThe DataRobot provider for Apache Airflow requires an environment with the following dependencies installed:Apache Airflow>= 2.3DataRobot Python API Client>= 3.2.0To install the DataRobot provider, run the following command:pipinstallairflow-provider-datarobotCreate a connection from Airflow to DataRobotThe next step is to create a connection from Airflow to DataRobot:In the Airflow user interface, clickAdmin > Connectionstoadd an Airflow connection.On theList Connectionpage, click+ Add a new record.In theAdd Connectiondialog box, configure the following fields:FieldDescriptionConnection Iddatarobot_default(this name is used by default in all operators)Connection TypeDataRobotAPI KeyA DataRobot API key, created in theDataRobot Developer Tools, from theAPI Keyssection.DataRobot endpoint URLhttps://app.datarobot.com/api/v2by defaultClickTestto establish a test connection between Airflow and DataRobot.When the connection test is successful, clickSave.Create preconfigured connections to DataRobotYou can create preconfigured connections to store and manage credentials to use with Airflow Operators,
replicating theconnection on the DataRobot side.Currently, the supported credential types are:CredentialsDescriptionDataRobot Basic CredentialsLogin/password pairsDataRobot GCP CredentialsGoogle Cloud Service account keyDataRobot AWS CredentialsAWS access keysDataRobot Azure Storage CredentialsAzure Storage secretDataRobot OAuth CredentialsOAuth tokensDataRobot JDBC DataSourceJDBC connection attributesAftercreating a preconfigured connection through the Airflow UI or API,
you can access your stored credentials withGetOrCreateCredentialOperatororGetOrCreateDataStoreOperatorto replicate them in DataRobot and retrieve the correspondingcredentials_idordatastore_id.JSON configuration for the DAG runOperators and sensors use parameters from theconfigJSON submitted when triggering the DAG; for example:{"training_data":"s3-presigned-url-or-local-path-to-training-data","project_name":"ProjectcreatedfromAirflow","autopilot_settings":{"target":"readmitted"},"deployment_label":"DeploymentcreatedfromAirflow","score_settings":{"intake_settings":{"type":"s3","url":"s3://path/to/scoring-data/Diabetes10k.csv","credential_id":"<credential_id>"},"output_settings":{"type":"s3","url":"s3://path/to/results-dir/Diabetes10k_predictions.csv","credential_id":"<credential_id>"}}}These config values are accessible in theexecute()method of any operator in the DAG
through thecontext["params"]variable; for example, to get training data, you could use the following:defexecute(self,context:Dict[str,Any])->str:...training_data=context["params"]["training_data"]...ModulesOperatorsGetOrCreateCredentialOperatorFetches a credential by name. This operator attempts to find a DataRobot credential with the provided name. If the credential doesn't exist, the operator creates it using the Airflow preconfigured connection with the same connection name.Returns a credential ID.Required config parameters:ParameterTypeDescriptioncredentials_param_namestrThe name of parameter in the config file for the credential name.GetOrCreateDataStoreOperatorFetches a DataStore by Connection name. If the DataStore does not exist, the operator attempts to create it using Airflow preconfigured connection with the same connection name.Returns a credential ID.Required config params:ParameterTypeDescriptionconnection_param_namestrThe name of the parameter in the config file for the connection name.CreateDatasetFromDataStoreOperatorLoads a dataset from a JDBC Connection to the DataRobot AI Catalog.Returns a dataset ID.Required config params:ParameterTypeDescriptiondatarobot_jdbc_connectionstrThe existing preconfigured DataRobot JDBC connection name.dataset_namestrThe name of the loaded dataset.table_schemastrThe database table schema.table_namestrThe source table name.do_snapshotboolIfTrue, creates a snapshot dataset. IfFalse, creates a remote dataset. If unset, uses the server default (True). Creating snapshots from non-file sources may be disabled by theDisable AI Catalog Snapshotspermission.persist_data_after_ingestionboolIfTrue, enforce saving all data (for download and sampling) and allow a user to view the extended data profile (which includes data statistics like min, max, median, mean, histogram, etc.). IfFalse, don't enforce saving data. If unset, uses the server default (True). The data schema (feature names and types) will still be available. Specifying this parameter toFalseanddoSnapshottoTrueresults in an error.UploadDatasetOperatorUploads a local file to the DataRobot AI Catalog.Returns a dataset ID.Required config params:ParameterTypeDescriptiondataset_file_pathstrThe local path to the training dataset.UpdateDatasetFromFileOperatorCreates a new dataset version from a file.Returns a dataset version ID when the new version uploads successfully.Required config params:ParameterTypeDescriptiondataset_idstrThe DataRobot AI Catalog dataset ID.dataset_file_pathstrThe local path to the training dataset.CreateDatasetVersionOperatorCreates a new version of the existing dataset in the AI Catalog.Returns a dataset version ID.Required config params:ParameterTypeDescriptiondataset_idstrThe DataRobot AI Catalog dataset ID.datasource_idstrThe existing DataRobot datasource ID.credential_idstrThe existing DataRobot credential ID.CreateOrUpdateDataSourceOperatorCreates a data source or updates it if it already exists.Returns a DataRobot DataSource ID.Required config params:ParameterTypeDescriptiondata_store_idstrTHe DataRobot datastore ID.CreateProjectOperatorCreates a DataRobot project.Returns a project ID.Several options of source dataset supported:Local file or pre-signed S3 URLCreate a project directly from a local file or a pre-signed S3 URL.Required config params:ParameterTypeDescriptiontraining_datastrThe pre-signed S3 URL or the local path to the training dataset.project_namestrThe project name.Note:In case of an S3 input, thetraining_datavalue must be apre-signed AWS S3 URL.AI Catalog dataset from config fileCreate a project from an existing dataset in the DataRobot AI Catalog using a dataset ID defined in the config file.Required config params:ParameterTypeDescriptiontraining_dataset_idstrThe dataset ID corresponding to existing dataset in the DataRobot AI Catalog.project_namestrThe project name.AI Catalog dataset from previous operatorCreate a project from an existing dataset in the DataRobot AI Catalog using a dataset ID from the previous operator.
In this case, your previous operator must return a valid dataset ID (for exampleUploadDatasetOperator)
and you should use this output value as adataset_idargument in theCreateProjectOperatorobject creation step.Required config params:ParameterTypeDescriptionproject_namestrThe project name.For moreproject settings, see the DataRobot documentation.TrainModelsOperatorRuns DataRobot Autopilot to train models.ReturnsNone.Parameters:ParameterTypeDescriptionproject_idstrThe DataRobot project ID.Required config params:ParameterTypeDescriptiontargetstrThe name of the column defining the modeling target."autopilot_settings":{"target":"readmitted"}For moreautopilot settings, see the DataRobot documentation.DeployModelOperatorDeploy a specified model.Returns a deployment ID.Parameters:ParameterTypeDescriptionmodel_idstrThe DataRobot model ID.Required config params:ParameterTypeDescriptiondeployment_labelstrThe deployment label name.For moredeployment settings, see the DataRobot documentation.DeployRecommendedModelOperatorDeploys a recommended model.Returns a deployment ID.Parameters:ParameterTypeDescriptionproject_idstrThe DataRobot project ID.Required config params:ParameterTypeDescriptiondeployment_labelstrThe deployment label name.For moredeployment settings, see the DataRobot documentation.ScorePredictionsOperatorScores batch predictions against the deployment.Returns a batch prediction job ID.Prerequisites:UseGetOrCreateCredentialOperatorto pass acredential_idfrom the preconfigured DataRobot Credentials (Airflow Connections) or manually set thecredential_idparameter in the config.Note:You canadd S3 credentials to DataRobot via the Python API client.Oruse a Dataset ID from the DataRobot AI Catalog.Oruse a DataStore ID for a JDBC source connection; you can useGetOrCreateDataStoreOperatorto passdatastore_idfrom a preconfigured Airflow Connection.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.intake_datastore_idstrThe DataRobot datastore ID for the JDBC source connection.output_datastore_idstrThe DataRobot datastore ID for the JDBC destination connection.intake_credential_idstrThe DataRobot credentials ID for the source connection.output_credential_idstrThe DataRobot credentials ID for the destination connection.Sample config: Pre-signed S3 URL"score_settings":{"intake_settings":{"type":"s3","url":"s3://my-bucket/Diabetes10k.csv",},"output_settings":{"type":"s3","url":"s3://my-bucket/Diabetes10k_predictions.csv",}}Sample config: Pre-signed S3 URL with a manually set credential ID"score_settings":{"intake_settings":{"type":"s3","url":"s3://my-bucket/Diabetes10k.csv","credential_id":"<credential_id>"},"output_settings":{"type":"s3","url":"s3://my-bucket/Diabetes10k_predictions.csv","credential_id":"<credential_id>"}}Sample config: Scoring dataset in the AI Catalog"score_settings":{"intake_settings":{"type":"dataset","dataset_id":"<datasetId>",},"output_settings":{}}For morebatch prediction settings, see the DataRobot documentation.GetTargetDriftOperatorGets the target drift from a deployment.Returns a dict with the target drift data.Parameters:ParameterTypeDescriptiondeployment_idstrTHe DataRobot deployment ID.No config params are required; however, theoptional paramsmay be passed in the config as follows:"target_drift":{}GetFeatureDriftOperatorGets the feature drift from a deployment.Returns a dict with the feature drift data.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.No config params are required; however, theoptional paramsmay be passed in the config as follows:"feature_drift":{}GetServiceStatsOperatorGets service stats measurements from a deployment.Returns a dict with the service stats measurements data.Parameters:ParameterTypeDescriptiondeployment_idstrTHe DataRobot deployment ID.No config params are required; however, theoptional paramsmay be passed in the config as follows:"service_stats":{}GetAccuracyOperatorGets the accuracy of a deployment’s predictions.Returns a dict with the accuracy for a deployment.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.No config params are required; however, theoptional paramsmay be passed in the config as follows:"accuracy":{}GetBiasAndFairnessSettingsOperatorGets the Bias And Fairness settings for deployment.Returns a dict with theBias And Fairness settings for a Deployment.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.No config params are required.UpdateBiasAndFairnessSettingsOperatorUpdates the Bias And Fairness settings for deployment.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.Sample config params:"protected_features": ["attribute1"],
"preferable_target_value": "True",
"fairness_metrics_set": "equalParity",
"fairness_threshold": 0.1,GetSegmentAnalysisSettingsOperatorGets the segment analysis settings for a deployment.Returns a dict with thesegment analysis settings for a deploymentParameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.No config params are required.UpdateSegmentAnalysisSettingsOperatorUpdates the segment analysis settings for a deployment.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.Sample config params:"segment_analysis_enabled": True,
"segment_analysis_attributes": ["attribute1", "attribute2"],GetMonitoringSettingsOperatorGets the monitoring settings for deployment.Returns a dict with the config params for a deployment.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.No config params are required.Sample monitoring settings:{
"drift_tracking_settings": {  } 
"association_id_settings": {  }
"predictions_data_collection_settings": {  }
}DictionaryDescriptiondrift_tracking_settingsThedrift tracking settings for this deployment.association_id_settingsTheassociation ID settings for this deployment.predictions_data_collection_settingsThepredictions data collection settings of this deployment.UpdateMonitoringSettingsOperatorUpdates monitoring settings for a deployment.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.Sample config params:"target_drift_enabled": True,
"feature_drift_enabled": True,
"association_id_column": ["id"],
"required_association_id": False,
"predictions_data_collection_enabled": False,BatchMonitoringOperatorCreates a batch monitoring job for the deployment.Returns a batch monitoring job ID.Prerequisites:UseGetOrCreateCredentialOperatorto pass acredential_idfrom the preconfigured DataRobot Credentials (Airflow Connections) or manually set thecredential_idparameter in the config.Note:You canadd S3 credentials to DataRobot via the Python API client.Oruse a Dataset ID from the DataRobot AI Catalog.Oruse a DataStore ID for a JDBC source connection; you can useGetOrCreateDataStoreOperatorto passdatastore_idfrom a preconfigured Airflow Connection.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.datastore_idstrThe DataRobot datastore ID.credential_idstrThe DataRobot credentials ID.Sample config params:Sample config"deployment_id":"61150a2fadb5586af4118980","monitoring_settings":{"intake_settings":{"type":"bigquery","dataset":"integration_example_demo","table":"actuals_demo","bucket":"datarobot_demo_airflow",},"monitoring_columns":{"predictions_columns":[{"class_name":"True","column_name":"target_True_PREDICTION"},{"class_name":"False","column_name":"target_False_PREDICTION"},],"association_id_column":"id","actuals_value_column":"ACTUAL",},}Sample config: Manually set credential ID"deployment_id":"61150a2fadb5586af4118980","monitoring_settings":{"intake_settings":{"type":"bigquery","dataset":"integration_example_demo","table":"actuals_demo","bucket":"datarobot_demo_airflow","credential_id":"<credential_id>"},"monitoring_columns":{"predictions_columns":[{"class_name":"True","column_name":"target_True_PREDICTION"},{"class_name":"False","column_name":"target_False_PREDICTION"},],"association_id_column":"id","actuals_value_column":"ACTUAL",},}For morebatch monitoring settings, see the DataRobot documentation.DownloadModelScoringCodeOperatorDownloads scoring code artifact from a model.Parameters:ParameterTypeDescriptionproject_idstrThe DataRobot project ID.model_idstrThe DataRobot model ID.base_pathstrThe base path for storing a downloaded model artifact.Sample config params:"source_code": False,For morescoring code download parameters, see the DataRobot documentation.DownloadDeploymentScoringCodeOperatorDownloads scoring code artifact from a deployment.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.base_pathstrThe base path for storing a downloaded model artifact.Sample config params:"source_code": False,
"include_agent": False,
"include_prediction_explanations": False,
"include_prediction_intervals": False,For morescoring code download parameters, see the DataRobot documentation.SubmitActualsFromCatalogOperatorDownloads scoring code artifact from a deployment.Returns an actuals upload job ID.Parameters:ParameterTypeDescriptiondeployment_idstrThe DataRobot deployment ID.dataset_idstrThe DataRobot AI Catalog dataset ID.dataset_version_idstrThe DataRobot AI Catalog dataset version ID.Sample config params:"association_id_column": "id",
"actual_value_column": "ACTUAL",
"timestamp_column": "timestamp",StartAutopilotOperatorTriggers DataRobot Autopilot to train set of models.Parameters:ParameterTypeDescriptionproject_idstrThe DataRobot project ID.featurelist_idstrSpecifies which feature list to use.relationships_configuration_idstrID of the relationships configuration to use.segmentation_task_idstrID of the relationships configuration to use.Sample config params:"autopilot_settings": {
    "target": "column_name",
    "mode": AUTOPILOT_MODE.QUICK,
}For moreanalyze_and_model parameters, see the DataRobot documentation.CreateExecutionEnvironmentOperatorCreate an execution environment.Returns an execution environment ID.Parameters:ParameterTypeDescriptionnamestrThe execution environment name.descriptionstrexecution environment description.programming_languagestrprogramming language of the environment to be created.Sample config params:"execution_environment_name": "My Demo Env",
"custom_model_description": "This is a custom model created by Airflow",
"programming_language": "python",For moreexecution environment creation parameters, see the DataRobot documentation.CreateExecutionEnvironmentVersionOperatorCreate an execution environment version.Returns a created execution environment version IDParameters:ParameterTypeDescriptionexecution_environment_idstrThe id of the execution environment.docker_context_pathstrThe path to a docker context archive or folder.environment_version_labelstrshort human readable string to label the version.environment_version_descriptionstrexecution environment version description.For moreexecution environment version creation parameters, see the DataRobot documentation.CreateCustomInferenceModelOperatorCreate a custom inference model.Returns a created custom model IDParameters:ParameterTypeDescriptionnamestrName of the custom model.descriptionstrDescription of the custom model.Sample DAG config params:"target_type": - Target type of the custom inference model.
    Values: [`datarobot.TARGET_TYPE.BINARY`, `datarobot.TARGET_TYPE.REGRESSION`,
    `datarobot.TARGET_TYPE.MULTICLASS`, `datarobot.TARGET_TYPE.UNSTRUCTURED`]
"target_name": - Target feature name.
    It is optional(ignored if provided) for `datarobot.TARGET_TYPE.UNSTRUCTURED` target type.
"programming_language": - Programming language of the custom learning model.
"positive_class_label": - Custom inference model positive class label for binary classification.
"negative_class_label": - Custom inference model negative class label for binary classification.
"prediction_threshold": - Custom inference model prediction threshold.
"class_labels": - Custom inference model class labels for multiclass classification.
"network_egress_policy": - Determines whether the given custom model is isolated, or can access the public network.
"maximum_memory": - The maximum memory that might be allocated by the custom-model.
"replicas": - A fixed number of replicas that will be deployed in the cluster.For morecustom inference model creation parameters, see the DataRobot documentation.CreateCustomModelVersionOperatorCreate a custom model version.Returns a created custom model version IDParameters:ParameterTypeDescriptioncustom_model_idstrThe ID of the custom model.base_environment_idstrThe ID of the base environment to use with the custom model version.training_dataset_idstrThe ID of the training dataset to assign to the custom model.holdout_dataset_idstrThe ID of the holdout dataset to assign to the custom model.custom_model_folderstrThe ID of the holdout dataset to assign to the custom model.create_from_previousboolif set to True - creates a custom model version containing files from a previous version.Sample DAG config params:"is_major_update" - The flag defining if a custom model version will be a minor or a major version.
"files" - The list of tuples, where values in each tuple are the local filesystem path and
            the path the file should be placed in the model.
"files_to_delete" - The list of a file items ids to be deleted.
"network_egress_policy": - Determines whether the given custom model is isolated, or can access the public network.
"maximum_memory": - The maximum memory that might be allocated by the custom-model.
"replicas": - A fixed number of replicas that will be deployed in the cluster.
"required_metadata_values" - Additional parameters required by the execution environment.
"keep_training_holdout_data" - If the version should inherit training and holdout data from the previous version.For morecustom inference model creation parameters, see the DataRobot documentation.CustomModelTestOperatorCreate and start a custom model test.Returns a created custom model test IDParameters:ParameterTypeDescriptioncustom_model_idstrThe ID of the custom model.custom_model_version_idstrThe ID of the custom model version.dataset_idstrThe id of the testing dataset for non-unstructured custom models. Ignored and not required for unstructured models.Sample DAG config params:"network_egress_policy": - Determines whether the given custom model is isolated, or can access the public network.
"maximum_memory": - The maximum memory that might be allocated by the custom-model.
"replicas": - A fixed number of replicas that will be deployed in the cluster.For morecustom model test creation parameters, see the DataRobot documentation.GetCustomModelTestOverallStatusOperatorGet a custom model testing overall status.Returns a custom model test overall statusParameters:ParameterTypeDescriptioncustom_model_test_idstrThe ID of the custom model test.For morecustom model test get status parameters, see the DataRobot documentation.CreateCustomModelDeploymentOperatorCreate a deployment from a DataRobot custom model image.Returns the created deployment idParameters:ParameterTypeDescriptioncustom_model_version_idstrThe id of the DataRobot custom model version to deploydeployment_namestra human readable label (name) of the deploymentdefault_prediction_server_idstran identifier of a prediction server to be used as the default prediction serverdescriptionstra human readable description of the deploymentimportancestrdeployment importanceFor morecreate_from_custom_model_version parameters, see the DataRobot documentation.GetDeploymentModelOperatorGets current model info from a deployment.Returns a model info from a DeploymentParameters:ParameterTypeDescriptiondeployment_idstrDataRobot deployment IDFor moreget deployment parameters, see the DataRobot documentation.ReplaceModelOperatorReplaces the current model for a deployment.Returns a model info from a DeploymentParameters:ParameterTypeDescriptiondeployment_idstrDataRobot deployment IDnew_model_idstrThe id of the new model to use. If replacing the deployment's model with a CustomInferenceModel, a specific CustomModelVersion ID must be used.reasonstrThe reason for the model replacement. Must be one of 'ACCURACY', 'DATA_DRIFT', 'ERRORS', 'SCHEDULED_REFRESH', 'SCORING_SPEED', or 'OTHER'. This value will be stored in the model history to keep track of why a model was replacedFor morereplace_model parameters, see the DataRobot documentation.ActivateDeploymentOperatorActivate or deactivate a Deployment.Returns the Deployment status (active/inactive)Parameters:ParameterTypeDescriptiondeployment_idstrDataRobot deployment IDactivatestrif set to True - activate deployment, if set to False - deactivate deploymentFor moreactivate deployment, see the DataRobot documentation.GetDeploymentStatusOperatorGet a Deployment status (active/inactive).Returns the Deployment status (active/inactive)Parameters:ParameterTypeDescriptiondeployment_idstrDataRobot deployment IDFor moredeployment, see the DataRobot documentation.RelationshipsConfigurationOperatorCreate a Relationships Configuration.Returns Relationships Configuration IDParameters:ParameterTypeDescriptiondataset_definitionsstrlist of dataset definitions. Each element is a dict retrieved from DatasetDefinitionOperator operatorrelationshipsstrlist of relationships. Each element is a dict retrieved from DatasetRelationshipOperator operatorfeature_discovery_settingsstrlist of feature discovery settings, optional. If not provided, it will be retrieved from DAG configuration params otherwise default settings will be used.For morefeature-discovery, see the DataRobot documentation.DatasetDefinitionOperatorCreate a Dataset definition for the Feature Discovery.Returns Dataset definition dictParameters:ParameterTypeDescriptiondataset_identifierstrAlias of the dataset (used directly as part of the generated feature names)dataset_idstrIdentifier of the dataset in DataRobot AI Catalogdataset_version_idstrIdentifier of the dataset version in DataRobot AI Catalogprimary_temporal_keystrName of the column indicating time of record creationfeature_list_idstrSpecifies which feature list to use.snapshot_policystrPolicy to use  when creating a project or making predictions. If omitted, by default endpoint will use 'latest'.For morecreate-dataset-definitions-and-relationships-using-helper-functions, see the DataRobot documentation.DatasetRelationshipOperatorCreate a Relationship between dataset defined in DatasetDefinition.Returns Dataset definition dictParameters:ParameterTypeDescriptiondataset1_identifierList[str]Identifier of the first dataset in this relationship. This is specified in the identifier field of dataset_definition structure. If None, then the relationship is with the primary dataset.dataset2_identifierList[str]Identifier of the second dataset in this relationship. This is specified in the identifier field of dataset_definition schema.dataset1_keysList[str]list of string (max length: 10 min length: 1). Column(s) from the first dataset which are used to join to the second datasetdataset2_keysList[str]list of string (max length: 10 min length: 1). Column(s) from the second dataset that are used to join to the first datasetfeature_derivation_window_startintHow many time units of each dataset's primary temporal key into the past relative to the datetimePartitionColumn the feature derivation window should begin. Will be a negative integer, If present, the feature engineering Graph will perform time-aware joins.feature_derivation_window_endintHow many time units of each dataset's record primary temporal key into the past relative to the datetimePartitionColumn the feature derivation window should end.  Will be a non-positive integer, if present. If present, the feature engineering Graph will perform time-aware joins.feature_derivation_window_time_unitstrTime unit of the feature derivation window. One ofdatarobot.enums.AllowedTimeUnitsSAFERIf present, time-aware joins will be used. Only applicable when dataset1_identifier is not provided.feature_derivation_windowsListList of feature derivation windows settings. If present, time-aware joins will be used. Only allowed when feature_derivation_window_start, feature_derivation_window_end and feature_derivation_window_time_unit are not provided.prediction_point_roundingList[dict]Closest value of prediction_point_rounding_time_unit to round the prediction point into the past when applying the feature deri if present.Only applicable when dataset1_identifier is not provided.prediction_point_rounding_time_unitstrTime unit of the prediction point rounding. One ofdatarobot.enums.AllowedTimeUnitsSAFEROnly applicable when dataset1_identifier is not provided.For morecreate-dataset-definitions-and-relationships-using-helper-functions, see the DataRobot documentation.ComputeFeatureImpactOperatorCreates Feature Impact job in DataRobot.Returns Feature Impact job IDParameters:ParameterTypeDescriptionproject_idstrDataRobot project IDmodel_idstrDataRobot model IDFor morerequest_feature_impact, see the DataRobot documentation.ComputeFeatureEffectsOperatorSubmit request to compute Feature Effects for the model.Returns Feature Effects job IDParameters:ParameterTypeDescriptionproject_idstrDataRobot project IDmodel_idstrDataRobot model IDFor morerequest_feature_impact, see the DataRobot documentation.ComputeShapOperatorSubmit request to compute SHAP impact job for the model.Returns SHAP impact job IDParameters:ParameterTypeDescriptionproject_idstrDataRobot project IDmodel_idstrDataRobot model IDFor moreshap-impact, see the DataRobot documentation.CreateExternalModelPackageOperatorCreate an external model package in DataRobot MLOps from JSON configuration.Returns a model package ID of newly created ModelPackage.Parameters:ParameterTypeDescriptionmodel_infostrA JSON object of external model parameters.Example of JSON configuration for a regression model:.. code-block:: python{
     "name": "Lending club regression",
     "modelDescription": {
             "description": "Regression on lending club dataset"
         }
     "target": {
         "type": "Regression",
         "name": "loan_amnt"
     }
}Example JSON for a binary classification model:.. code-block:: python{
    "name": "Surgical Model",
    "modelDescription": {
        "description": "Binary classification on surgical dataset",
        "location": "/tmp/myModel"
        },
        "target": {
            "type": "Binary",
            "name": "complication",
            "classNames": ["Yes","No"],  # minority/positive class should be listed first
            "predictionThreshold": 0.5
        }
    }
}Example JSON for a multiclass classification model:.. code-block:: python{
    "name": "Iris classifier",
    "modelDescription": {
    "description": "Classification on iris dataset",
    "location": "/tmp/myModel"
},
    "target": {
        "type": "Multiclass",
        "name": "Species",
        "classNames": [
            "Iris-versicolor",
            "Iris-virginica",
            "Iris-setosa"
        ]
    }
}DeployModelPackageOperatorCreate a deployment from a DataRobot model package.Returns The created deployment IDParameters:ParameterTypeDescriptiondeployment_namestrA human readable label of the deployment.model_package_idstrThe ID of the DataRobot model package to deploy.default_prediction_server_idstrAn identifier of a prediction server to be used as the default prediction server. When working with prediction environments, default prediction server Id should not be providedprediction_environment_idstrAn identifier of a prediction environment to be used for model deployment.descriptionstrA human readable description of the deployment.importancestrDeployment importance level.user_provided_idstrA user-provided unique ID associated with a deployment definition in a remote git repository.additional_metadataDict[str, str]A Key/Value pair dict, with additional metadata.AddExternalDatasetOperatorUpload a new dataset from a catalog dataset to make predictions for a modelReturns external dataset ID for the modelParameters:ParameterTypeDescriptionproject_idstrDataRobot project IDdataset_idstrDataRobot AI Catalog dataset IDcredential_idstrDataRobot Credentials IDdataset_version_idstrDataRobot AI Catalog dataset version IDFor moreupload_dataset_from_catalog, see the DataRobot documentation.RequestModelPredictionsOperatorRequests predictions against a previously uploaded dataset.Returns model predictions job IDParameters:ParameterTypeDescriptionproject_idstrDataRobot project IDmodel_idstrDataRobot model IDexternal_dataset_idstrDataRobot external dataset IDFor morerequest_predictions, see the DataRobot documentation.TrainModelOperatorSubmit a job to the queue to train a model from specific blueprint.Returns model training job IDParameters:ParameterTypeDescriptionproject_idstrDataRobot project IDblueprint_idstrDataRobot blueprint IDfeaturelist_idstrThe identifier of the featurelist to use. If not defined, the default for this project is used.source_project_idstrWhich project created this blueprint_id. IfNone, it defaults to looking in this project. Note that you must have read permissions in this project.Example of DAG config params:
{
"sample_pct":
"scoring_type":
"training_row_count":
"n_clusters":
}For morestart-training-a-model, see the DataRobot documentation.RetrainModelOperatorSubmit a job to the queue to retrain a model on a specific sample size and/or custom featurelist.Returns a model retraining job IDParameters:ParameterTypeDescriptionproject_idstrDataRobot project IDmodel_idstrDataRobot model IDfeaturelist_idstrThe identifier of the featurelist to use. If not defined, the default for this project is used.Example of DAG config params:
{
"sample_pct":
"scoring_type":
"training_row_count":
}For moretrain-a-model-on-a-different-sample-size, see the DataRobot documentation.PredictionExplanationsInitializationOperatorTriggering a prediction explanations initialization of a model.Returns a Prediction Explanations Initialization job IDParameters:ParameterTypeDescriptionproject_idstrDataRobot project IDmodel_idstrDataRobot model IDFor moreprediction-explanations, see the DataRobot documentation.ComputePredictionExplanationsOperatorCreate prediction explanations for the specified dataset.Returns a Triggered prediction explanations for the specified dataset job IDParameters:ParameterTypeDescriptionproject_idstrDataRobot project IDmodel_idstrDataRobot model IDexternal_dataset_idstrDataRobot external dataset IDExample of DAG config params:{
"max_explanations"
"threshold_low"
"threshold_high"
}For moreprediction-explanations, see the DataRobot documentation.SensorsAutopilotCompleteSensorChecks if Autopilot is complete.Parameters:ParameterTypeDescriptionproject_idstrThe DataRobot project ID.ScoringCompleteSensorChecks if batch scoring is complete.Parameters:ParameterTypeDescriptionjob_idstrThe batch prediction job ID.MonitoringJobCompleteSensorChecks if a monitoring job is complete.Parameters:ParameterTypeDescriptionjob_idstrThe batch monitoring job ID.BaseAsyncResolutionSensorChecks if the DataRobot Async API call is complete.Parameters:ParameterTypeDescriptionjob_idstrThe DataRobot async API call status check ID.DataRobotJobSensorChecks whether DataRobot Job is complete.Parameters:ParameterTypeDescriptionproject_idstrDataRobot project ID.job_idstrDataRobot Job ID.ModelTrainingJobSensorChecks whether DataRobot Model Training Job is complete.Returns False if job not yet completed, PokeReturnValue(True, trained_model.id) if model training completedParameters:ParameterTypeDescriptionproject_idstrDataRobot project ID.job_idstrDataRobot Job ID.HooksDataRobotHookA hook to initialize the DataRobot Public API client.PipelineThe modules described above allow you to construct a standard DataRobot pipeline in an Airflow DAG:create_project_op >> train_models_op >> autopilot_complete_sensor >> deploy_model_op >> score_predictions_op >> scoring_complete_sensorExample DAGSSee the thedatarobot_provider/example_dagsdirectory for the example DAGs.You can find the following examples using a preconfigured connection in thedatarobot_provider/example_dagsdirectory:Example DAGDescriptiondatarobot_pipeline_dag.pyRun the basic end-to-end workflow in DataRobot.datarobot_score_dag.pyPerform DataRobot batch scoring.datarobot_jdbc_batch_scoring_dag.pyPerform DataRobot batch scoring with a JDBC data source.datarobot_aws_s3_batch_scoring_dag.pyUse DataRobot AWS Credentials withScorePredictionsOperator.datarobot_gcp_storage_batch_scoring_dag.pyUse DataRobot GCP Credentials withScorePredictionsOperator.datarobot_bigquery_batch_scoring_dag.pyUse DataRobot GCP Credentials withScorePredictionsOperator.datarobot_azure_storage_batch_scoring_dag.pyUse DataRobot Azure Storage Credentials withScorePredictionsOperator.datarobot_jdbc_dataset_dag.pyUpload a dataset to the AI Catalog through a JDBC connection.datarobot_batch_monitoring_job_dag.pyRun a batch monitoring job.datarobot_create_project_from_ai_catalog_dag.pyCreate a DataRobot project from a DataRobot AI Catalog dataset.datarobot_create_project_from_dataset_version_dag.pyCreate a DataRobot project from a specific dataset version in the DataRobot AI Catalog.datarobot_dataset_new_version_dag.pyCreate a new version of an existing dataset in the DataRobot AI Catalog.datarobot_dataset_upload_dag.pyUpload a local file to the DataRobot AI Catalog.datarobot_get_datastore_dag.pyCreate a DataRobot DataStore withGetOrCreateDataStoreOperator.datarobot_jdbc_dataset_dag.pyCreate a DataRobot project from a JDBC data source.datarobot_jdbc_dynamic_dataset_dag.pyCreate a DataRobot project from a JDBC dynamic data source.datarobot_upload_actuals_catalog_dag.pyUpload actuals from the DataRobot AI Catalog.deployment_service_stats_dag.pyGet a deployment's service statistics withGetServiceStatsOperatordeployment_stat_and_accuracy_dag.pyGet a deployment's service statistics and accuracy.deployment_update_monitoring_settings_dag.pyUpdate a deployment's monitoring settings.deployment_update_segment_analysis_settings_dag.pyUpdate a deployment's segment analysis settings.download_scoring_code_from_deployment_dag.pyDownload scoring code (JAR file) from a DataRobot deployment.advanced_datarobot_pipeline_jdbc_dag.pyRun the advanced end-to-end workflow in DataRobot.datarobot_autopilot_options_pipeline_dag.pyCreates datarobot project and starts autopilot with advanced options.datarobot_custom_model_pipeline_dag.pyCreating end-to-end workflow with custom models in DataRobot.datarobot_custom_partitioning_pipeline_dag.pyCreating custom partitioned project, train models piplinedatarobot_datetime_partitioning_pipeline_dag.pyCreating datetime partitioned project, train models pipline.datarobot_external_model_pipeline_dag.pyCreating end-to-end workflow with external models in DataRobot.datarobot_feature_discovery_pipeline_dag.pyCreating feature-discovery DataRobot project, train models pipline.datarobot_timeseries_pipeline_dag.pyCreating timeseries DataRobot project, train models pipline.deployment_activate_deactivate_dag.pyExample of Deployment activation/deactivaion and get Deployment status.deployment_replace_model_dag.pyExample of Deployment model replacement.model_compute_insights_dag.pyExample of compute FeatureImpact and FeatureEffects job.model_compute_prediction_explanations_dag.pyExample of compute prediction explanations job.model_compute_predictions_dag.pyExample of compute predictions for model.model_compute_shap_dag.pyExample of compute SHAP job.model_retrain_dag.pyExample of model retraining job on specific sample size/featurelist.model_train_dag.pyExample of model training job based on specific blueprint.The advanced end-to-end workflow in DataRobot (advanced_datarobot_pipeline_jdbc_dag.py) contains the following steps:Ingest a dataset to the AI Catalog from JDBC datasourceCreate a DataRobot projectTrain models using AutopilotDeploy the recommended modelChange deployment settings (enable monitoring settings, segment analysis, and bias and fairness)Run batch scoring using a JDBC datasourceUpload actuals from a JDBC datasourceCollect deployment metrics: service statistics, features drift, target drift, accuracy and process it with custom python operator.IssuesPlease submitissuesandpull requestsin our official repo:https://github.com/datarobot/airflow-provider-datarobotWe are happy to hear from you. Please email any feedback to the authors atsupport@datarobot.com.Copyright NoticeCopyright 2023 DataRobot, Inc. and its affiliates.All rights reserved.This is proprietary source code of DataRobot, Inc. and its affiliates.Released under the terms of DataRobot Tool and Utility Agreement.