Repository: databricks/terraform-provider-databricks
Pull request #4547: [Fix] handle auto-enabled errors with`databricks_system_schema`
Author: login: nkvuong
Created: 2025-03-04T08:34:10Z | Merged: 2025-03-12T06:40:22Z

## Changes
- Handle new error message `billing system schema can only be enabled by Databricks`. Resolves #4539
- Update documentation with warnings about auto-enabled system schemas.
- Enable system schema integration test on GCP

## Tests
<!-- 
How is this tested? Please see the checklist below and also describe any other relevant tests 
-->

- [x] `make test` run locally
- [x] relevant change in `docs/` folder
- [x] covered with integration tests in `internal/acceptance`
- [x] using Go SDK
- [x] using TF Plugin Framework


Linked issue #4539: [ISSUE] Failures for Auto-Enabled System Tables
While using Terraform with Databricks, we noticed state changes related to system tables during the terraform plan step. The following system tables are affected:

- databricks_system_schema.access_logs is tainted and requires replacement.
- databricks_system_schema.billing_logs is marked for creation.
- databricks_system_schema.compute_logs is tainted and requires replacement.
- databricks_system_schema.lakeflow_logs is tainted and requires replacement.

However, during the `terraform apply` step, encounter the following error, which prevents the deployment of new changes:

```
Error: cannot create system schema: billing system schema can only be enabled by Databricks.

 with databricks_system_schema.billing_logs,
 on main.tf line 226, in resource "databricks_system_schema" "billing_logs":
 226: resource "databricks_system_schema" "billing_logs" {

Error: Process completed with exit code 1.
```

### Root Cause:

The `billing` schema is now enabled by default in Databricks, meaning that any resource block attempting to create it manually will result in an error.

### Proposed Solution:

Documentation Update: Clearly state in the documentation that certain system schemas (such as `billing`) may be auto-enabled and should not be manually declared in Terraform configurations.

Error Handling Improvement: Instead of failing the terraform apply process, introduce logic that detects and ignores errors when certain system tables are already enabled by default.

Terraform Provider Enhancement: Modify the Databricks Terraform provider to automatically detect auto-enabled system schemas and exclude them from the state file or ignore changes during `terraform apply`.

Diff:
diff --git a/NEXT_CHANGELOG.md b/NEXT_CHANGELOG.md
index 2d10a0c1f6..a8af628928 100644
--- a/NEXT_CHANGELOG.md
+++ b/NEXT_CHANGELOG.md
@@ -6,7 +6,8 @@
 
 ### Bug Fixes
 
- - Skip Read after Create in `databricks_secret_acl` to avoid errors([#4548](https://github.com/databricks/terraform-provider-databricks/pull/4548)).
+ * Handle auto-enabled errors with `databricks_system_schema` [#4547](https://github.com/databricks/terraform-provider-databricks/pull/4547)
+ * Skip Read after Create in `databricks_secret_acl` to avoid errors([#4548](https://github.com/databricks/terraform-provider-databricks/pull/4548)).
 
 ### Documentation
 
diff --git a/catalog/resource_system_schema.go b/catalog/resource_system_schema.go
index 6c033fab89..e58afaae30 100644
--- a/catalog/resource_system_schema.go
+++ b/catalog/resource_system_schema.go
@@ -21,6 +21,10 @@ func ResourceSystemSchema() common.Resource {
 			Type:     schema.TypeString,
 			Computed: true,
 		}
+		m["auto_enabled"] = &schema.Schema{
+			Type:     schema.TypeBool,
+			Computed: true,
+		}
 		m["state"].Computed = true
 		return m
 	})
@@ -49,12 +53,20 @@ func ResourceSystemSchema() common.Resource {
 			MetastoreId: metastoreSummary.MetastoreId,
 			SchemaName:  new,
 		})
-		//ignore "schema <schema-name> already exists" error
-		if err != nil && !strings.Contains(err.Error(), "already exists") {
-			return err
+		if err != nil {
+			//ignore "<schema-name> system schema can only be enabled by Databricks" error, also mark it to make delete no-op
+			d.Set("auto_enabled", strings.Contains(err.Error(), "can only be enabled by Databricks"))
+			//ignore "schema <schema-name> already exists" error
+			if !strings.Contains(err.Error(), "already exists") && !strings.Contains(err.Error(), "can only be enabled by Databricks") {
+				return err
+			}
 		}
 		//disable old schemas if needed
 		if old != "" {
+			if d.Get("auto_enabled").(bool) {
+				log.Printf("[WARN] %s is auto enabled, ignoring it", old)
+				return nil
+			}
 			err = w.SystemSchemas.Disable(ctx, catalog.DisableRequest{
 				MetastoreId: metastoreSummary.MetastoreId,
 				SchemaName:  old,
@@ -116,6 +128,10 @@ func ResourceSystemSchema() common.Resource {
 			if err != nil {
 				return err
 			}
+			if d.Get("auto_enabled").(bool) {
+				log.Printf("[WARN] %s is auto enabled, ignoring it", schemaName)
+				return nil
+			}
 			w, err := c.WorkspaceClient()
 			if err != nil {
 				return err
diff --git a/catalog/resource_system_schema_test.go b/catalog/resource_system_schema_test.go
index 8973d8727a..602ad3879c 100644
--- a/catalog/resource_system_schema_test.go
+++ b/catalog/resource_system_schema_test.go
@@ -1,13 +1,16 @@
 package catalog
 
 import (
+	"errors"
 	"net/http"
 	"testing"
 
+	"github.com/databricks/databricks-sdk-go/experimental/mocks"
 	"github.com/databricks/databricks-sdk-go/service/catalog"
 	"github.com/databricks/terraform-provider-databricks/common"
 	"github.com/databricks/terraform-provider-databricks/qa"
 	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/mock"
 )
 
 func TestSystemSchemaCreate(t *testing.T) {
@@ -87,6 +90,40 @@ func TestSystemSchemaCreate_Error(t *testing.T) {
 	assert.Equal(t, "", d.Id(), "Id should be empty for error creates")
 }
 
+func TestSystemSchemaCreateAlreadyEnabled(t *testing.T) {
+	qa.ResourceFixture{
+		MockWorkspaceClientFunc: func(w *mocks.MockWorkspaceClient) {
+			w.GetMockMetastoresAPI().EXPECT().Summary(mock.Anything).Return(&catalog.GetMetastoreSummaryResponse{
+				MetastoreId: "abc",
+			}, nil)
+			e := w.GetMockSystemSchemasAPI().EXPECT()
+			e.Enable(mock.Anything, catalog.EnableRequest{
+				MetastoreId: "abc",
+				SchemaName:  "billing",
+			}).Return(errors.New("billing system schema can only be enabled by Databricks"))
+			e.ListByMetastoreId(mock.Anything, "abc").Return(&catalog.ListSystemSchemasResponse{
+				Schemas: []catalog.SystemSchemaInfo{
+					{
+						Schema: "access",
+						State:  catalog.SystemSchemaInfoStateEnableCompleted,
+					},
+					{
+						Schema: "billing",
+						State:  catalog.SystemSchemaInfoStateEnableCompleted,
+					},
+				},
+			}, nil)
+		},
+		Resource: ResourceSystemSchema(),
+		HCL:      `schema = "billing"`,
+		Create:   true,
+	}.ApplyAndExpectData(t, map[string]any{
+		"schema":    "billing",
+		"id":        "abc|billing",
+		"full_name": "system.billing",
+	})
+}
+
 func TestSystemSchemaUpdate(t *testing.T) {
 	d, err := qa.ResourceFixture{
 		Fixtures: []qa.HTTPFixture{
diff --git a/catalog/system_schema_test.go b/catalog/system_schema_test.go
index aadd89105b..91ce03b73f 100644
--- a/catalog/system_schema_test.go
+++ b/catalog/system_schema_test.go
@@ -1,20 +1,19 @@
 package catalog_test
 
 import (
-	"os"
 	"testing"
 
 	"github.com/databricks/terraform-provider-databricks/internal/acceptance"
 )
 
 func TestUcAccResourceSystemSchema(t *testing.T) {
-	if os.Getenv("GOOGLE_CREDENTIALS") != "" {
-		t.Skipf("databricks_system_schema resource not available on GCP")
-	}
 	acceptance.UnityWorkspaceLevel(t, acceptance.Step{
 		Template: `
-		resource "databricks_system_schema" "this" {
+		resource "databricks_system_schema" "access" {
 			schema = "access"
+		}
+		resource "databricks_system_schema" "billing" {
+			schema = "billing"
 		}`,
 	})
 }
diff --git a/docs/resources/system_schema.md b/docs/resources/system_schema.md
index 02945703f9..c1aa3bb965 100644
--- a/docs/resources/system_schema.md
+++ b/docs/resources/system_schema.md
@@ -7,6 +7,8 @@ subcategory: "Unity Catalog"
 
 -> This resource can only be used with a workspace-level provider!
 
+-> Certain system schemas (such as `billing`) may be auto-enabled once GA and should not be manually declared in Terraform configurations.
+
 Manages system tables enablement. System tables are a Databricks-hosted analytical store of your account’s operational data. System tables can be used for historical observability across your account. System tables must be enabled by an account admin.
 
 ## Example Usage


Reviews:
user: login: alexott
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T07:15:20Z
user: login: nkvuong
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T07:32:25Z
user: login: alexott
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T16:10:25Z

Review comments:
diff_hunk: @@ -9,6 +9,8 @@ subcategory: "Unity Catalog"
 
 Manages system tables enablement. System tables are a Databricks-hosted analytical store of your account’s operational data. System tables can be used for historical observability across your account. System tables must be enabled by an account admin.
 
+Certain system schemas (such as `billing`) may be auto-enabled once GA and should not be manually declared in Terraform configurations.
path: docs/resources/system_schema.md
user: login: alexott
body: Should we put this into the callout?
created_at: 2025-03-05T07:15:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -9,6 +9,8 @@ subcategory: "Unity Catalog"
 
 Manages system tables enablement. System tables are a Databricks-hosted analytical store of your account’s operational data. System tables can be used for historical observability across your account. System tables must be enabled by an account admin.
 
+Certain system schemas (such as `billing`) may be auto-enabled once GA and should not be manually declared in Terraform configurations.
path: docs/resources/system_schema.md
user: login: nkvuong
body: done
created_at: 2025-03-05T07:32:24Z
author_association: CONTRIBUTOR

PR comments:
user: login: alexott
created_at: 2025-03-04T16:47:34Z
body: @nkvuong how it will handle destroy? Will it fail trying to disable the schema?
author_association: CONTRIBUTOR
user: login: nkvuong
created_at: 2025-03-05T07:19:23Z
body: > @nkvuong how it will handle destroy? Will it fail trying to disable the schema?

@alexott our API actually succeeds 😕 

author_association: CONTRIBUTOR
user: login: alexott
created_at: 2025-03-05T07:35:28Z
body: @nkvuong maybe we can add a computed flag that specifies that it's auto-enabled? And then in the `Delete` skip the deletion?
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-03-11T19:08:27Z
body: <!-- INTEGRATION_TESTS_MANUAL -->
If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
[go/deco-tests-run/terraform](https://go/deco-tests-run/terraform)

Inputs:
* PR number: 4547
* Commit SHA: `d127b8c8183eb483a892627d006355706e9e66be`

Checks will be approved automatically on success.

author_association: CONTRIBUTOR