Terraformを用いたエンドツーエンドのDatabricksワークスペース管理

[Manage a workspace end\-to\-end using Terraform \| Databricks on AWS](https://docs.databricks.com/dev-tools/terraform/workspace-management.html) [2021/12/28時点]の翻訳です。

:::note warn
本書は抄訳であり内容の正確性を保証するものではありません。正確な内容に関しては原文を参照ください。
:::

:::note info
**実験段階**
Databricks TerraformプロバイダーはDatabricksやAWSに正式にサポートはされていません。これは、Databricksのフィールドエンジニアリングチームによってメンテナンスされ、as-isで提供されています。サービスレベルアグリーメント(SLA)は存在せず、AWSはいかなる保証も行いません。プロバイダーに問題を見つけた場合には、[GitHub Issue](https://github.com/databrickslabs/terraform-provider-databricks/issues/new/choose)を起票することで、時間が許す限りプロジェクトのメンテナーによってレビューされます。
:::

本書では、[Databricks Terraformプロバイダー](https://qiita.com/taka_yayoi/items/691ee4909b0961de5ad1)を用いた[Databricksワークスペース](https://qiita.com/taka_yayoi/items/11d4ef49871e9b1292bb)のリソース管理方法を説明します。

以下の設定ブロックは最も一般的な変数、[databricks\_spark\_version](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/data-sources/spark_version)、[databricks\_node\_type](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/data-sources/node_type)、[databricks\_current\_user](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/data-sources/current_user)を初期化します。

```
terraform {
  required_providers {
    databricks = {
      source  = "databrickslabs/databricks"
    }
  }
}

provider "databricks" {}

data "databricks_current_user" "me" {}
data "databricks_spark_version" "latest" {}
data "databricks_node_type" "smallest" {
  local_disk = true
}
```

# 標準機能

これらのリソースには管理者権限は不要です。より詳細なドキュメントに関しては専門のページ、[databricks\_secret\_scope](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/secret_scope)、[databricks\_secret](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/secret)、[databricks\_notebook](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/data-sources/notebook)、[databricks\_job](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/job)、[databricks\_cluster](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/cluster)、[databricks\_cluster\_policy](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/cluster_policy)、[databricks\_instance\_pool](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/instance_pool)をご覧ください。

```
resource "databricks_secret_scope" "this" {
  name = "demo-${data.databricks_current_user.me.alphanumeric}"
}

resource "databricks_token" "pat" {
  comment          = "Created from ${abspath(path.module)}"
  lifetime_seconds = 3600
}

resource "databricks_secret" "token" {
  string_value = databricks_token.pat.token_value
  scope        = databricks_secret_scope.this.name
  key          = "token"
}

resource "databricks_notebook" "this" {
  path     = "${data.databricks_current_user.me.home}/Terraform"
  language = "PYTHON"
  content_base64 = base64encode(<<-EOT
    token = dbutils.secrets.get('${databricks_secret_scope.this.name}', '${databricks_secret.token.key}')
    print(f'This should be redacted: {token}')
    EOT
  )
}

resource "databricks_job" "this" {
  name = "Terraform Demo (${data.databricks_current_user.me.alphanumeric})"
  new_cluster {
    num_workers   = 1
    spark_version = data.databricks_spark_version.latest.id
    node_type_id  = data.databricks_node_type.smallest.id
  }

  notebook_task {
    notebook_path = databricks_notebook.this.path
  }

  email_notifications {}
}

resource "databricks_cluster" "this" {
  cluster_name = "Exploration (${data.databricks_current_user.me.alphanumeric})"
  spark_version           = data.databricks_spark_version.latest.id
  instance_pool_id        = databricks_instance_pool.smallest_nodes.id
  autotermination_minutes = 20
  autoscale {
    min_workers = 1
    max_workers = 10
  }
}

resource "databricks_cluster_policy" "this" {
  name = "Minimal (${data.databricks_current_user.me.alphanumeric})"
  definition = jsonencode({
    "dbus_per_hour" : {
      "type" : "range",
      "maxValue" : 10
    },
    "autotermination_minutes" : {
      "type" : "fixed",
      "value" : 20,
      "hidden" : true
    }
  })
}

resource "databricks_instance_pool" "smallest_nodes" {
  instance_pool_name = "Smallest Nodes (${data.databricks_current_user.me.alphanumeric})"
  min_idle_instances = 0
  max_capacity       = 30
  node_type_id       = data.databricks_node_type.smallest.id
  preloaded_spark_versions = [
    data.databricks_spark_version.latest.id
  ]

  idle_instance_autotermination_minutes = 20
}

output "notebook_url" {
  value = databricks_notebook.this.url
}

output "job_url" {
  value = databricks_job.this.url
}
```

# ワークスペースのセキュリティ

セキュリティの管理には管理者権限が必要となります。詳細なドキュメントに関しては専用のページ、[databricks\_secret\_acl](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/secret_acl)、[databricks\_group](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/data-sources/group)、[databricks\_user](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/user)、[databricks\_group\_member](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/group_member)、[databricks\_permissions](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/permissions)をご覧下さい。

```
resource "databricks_secret_acl" "spectators" {
  principal  = databricks_group.spectators.display_name
  scope      = databricks_secret_scope.this.name
  permission = "READ"
}

resource "databricks_group" "spectators" {
  display_name = "Spectators (by ${data.databricks_current_user.me.alphanumeric})"
}

resource "databricks_user" "dummy" {
  user_name    = "dummy+${data.databricks_current_user.me.alphanumeric}@example.com"
  display_name = "Dummy ${data.databricks_current_user.me.alphanumeric}"
}

resource "databricks_group_member" "a" {
  group_id  = databricks_group.spectators.id
  member_id = databricks_user.dummy.id
}

resource "databricks_permissions" "notebook" {
  notebook_path = databricks_notebook.this.id
  access_control {
    user_name        = databricks_user.dummy.user_name
    permission_level = "CAN_RUN"
  }
  access_control {
    group_name       = databricks_group.spectators.display_name
    permission_level = "CAN_READ"
  }
}

resource "databricks_permissions" "job" {
  job_id = databricks_job.this.id
  access_control {
    user_name        = databricks_user.dummy.user_name
    permission_level = "IS_OWNER"
  }
  access_control {
    group_name       = databricks_group.spectators.display_name
    permission_level = "CAN_MANAGE_RUN"
  }
}

resource "databricks_permissions" "cluster" {
  cluster_id = databricks_cluster.this.id
  access_control {
    user_name        = databricks_user.dummy.user_name
    permission_level = "CAN_RESTART"
  }
  access_control {
    group_name       = databricks_group.spectators.display_name
    permission_level = "CAN_ATTACH_TO"
  }
}

resource "databricks_permissions" "policy" {
  cluster_policy_id = databricks_cluster_policy.this.id
  access_control {
    group_name       = databricks_group.spectators.display_name
    permission_level = "CAN_USE"
  }
}

resource "databricks_permissions" "pool" {
  instance_pool_id = databricks_instance_pool.smallest_nodes.id
  access_control {
    group_name       = databricks_group.spectators.display_name
    permission_level = "CAN_ATTACH_TO"
  }
}
```

# ストレージ

要件に応じて、以下のことを行うことができます。

- [databricks\_dbfs\_file](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/dbfs_file)リソースによるJAR、Wheel、Eggライブラリの管理
- [databricks\_dbfs\_file\_paths](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/data-sources/dbfs_file_paths)データソースによるDBFS上のエントリーの一覧
- [databricks\_dbfs\_file](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/data-sources/dbfs_file)データソースによる小規模ファイルの中身の取得
- [databricks\_aws\_s3\_mount](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/aws_s3_mount)リソースを用いたAWSストレージのマウント

# 高度な設定

[databricks\_workspace\_conf](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/workspace_conf)、[databricks\_ip\_access\_list](https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/ip_access_list)リソースの専用ページをご覧ください。

```
data "http" "my" {
  url = "https://ifconfig.me"
}

resource "databricks_workspace_conf" "this" {
  custom_config = {
    "enableIpAccessLists": "true"
  }
}

resource "databricks_ip_access_list" "only_me" {
  label = "only ${data.http.my.body} is allowed to access workspace"
  list_type = "ALLOW"
  ip_addresses = ["${data.http.my.body}/32"]
  depends_on = [databricks_workspace_conf.this]
}
```

### Databricks 無料トライアル

[Databricks 無料トライアル](https://databricks.com/jp/try-databricks)