Repository: pulumi/pulumi-terraform-bridge
Pull request #1355 (2023-09-06 19:40:05)

Introduce a hook for editing state (outputs)
When working on https://github.com/pulumi/pulumi-aws/issues/2778 I would like to be able to remove "tagsAll" from Pulumi state file before it gets persisted, although the underlying TF provider keeps populating this output property. Unfortunately it's really non-trivial to do this in the provider itself without changes to the bridge. The proposed change here introduces a hook for editing the state right before it gets sent to the engine which includes these gRPC methods:

- Create
- Update
- Read

In particular the hook edits the `outputs` bits of the state. The `inputs` bits can be normally modified in PreCheckCallback, although there is no mechanism yet to modify them in the Read scenario (pulumi refresh or import). It's left for future work as not really required by the scenario being addressed here. 



t0yv0: Going to throw a few unit tests in here.
github-actions[bot]: Diff for [pulumi-random](https://github.com/pulumi-bot/pulumi-random/compare/c0fdfde..9bd7922) with merge commit 66bf94e54d8914b74dfa56cbaf2ea5e641cbb165
github-actions[bot]: Diff for [pulumi-azuread](https://github.com/pulumi-bot/pulumi-azuread/compare/a257ab3..0638436) with merge commit 66bf94e54d8914b74dfa56cbaf2ea5e641cbb165
github-actions[bot]: Diff for [pulumi-random](https://github.com/pulumi-bot/pulumi-random/compare/9bdf670..1f613df) with merge commit 675b1e5453839314a5c3a7b6337f2465f30d6b07
github-actions[bot]: Diff for [pulumi-azuread](https://github.com/pulumi-bot/pulumi-azuread/compare/c81c475..494f48c) with merge commit 675b1e5453839314a5c3a7b6337f2465f30d6b07
github-actions[bot]: Diff for [pulumi-random](https://github.com/pulumi-bot/pulumi-random/compare/0fead1b..b5f58ea) with merge commit afcadbb1c663f2bdd37bba3a79f30c2225407f57
github-actions[bot]: Diff for [pulumi-azuread](https://github.com/pulumi-bot/pulumi-azuread/compare/0e3d7e2..fa533d3) with merge commit afcadbb1c663f2bdd37bba3a79f30c2225407f57
codecov-commenter: ## [Codecov](https://app.codecov.io/gh/pulumi/pulumi-terraform-bridge/pull/1355?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pulumi) Report
> Merging [#1355](https://app.codecov.io/gh/pulumi/pulumi-terraform-bridge/pull/1355?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pulumi) (8fc3663) into [master](https://app.codecov.io/gh/pulumi/pulumi-terraform-bridge/commit/5b11d75dbcf38b53f31e4cf046f2df5f8798377e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pulumi) (5b11d75) will **increase** coverage by `0.00%`.
> The diff coverage is `50.00%`.

```diff
@@           Coverage Diff           @@
##           master    #1355   +/-   ##
=======================================
  Coverage   55.91%   55.91%           
=======================================
  Files         146      145    -1     
  Lines       24277    24293   +16     
=======================================
+ Hits        13574    13583    +9     
- Misses       9734     9738    +4     
- Partials      969      972    +3     
```


| [Files Changed](https://app.codecov.io/gh/pulumi/pulumi-terraform-bridge/pull/1355?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pulumi) | Coverage Δ | |
|---|---|---|
| [pkg/tfbridge/info.go](https://app.codecov.io/gh/pulumi/pulumi-terraform-bridge/pull/1355?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pulumi#diff-cGtnL3RmYnJpZGdlL2luZm8uZ28=) | `36.76% <ø> (ø)` | |
| [pkg/tfbridge/provider.go](https://app.codecov.io/gh/pulumi/pulumi-terraform-bridge/pull/1355?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pulumi#diff-cGtnL3RmYnJpZGdlL3Byb3ZpZGVyLmdv) | `46.42% <50.00%> (+0.07%)` | :arrow_up: |

... and [1 file with indirect coverage changes](https://app.codecov.io/gh/pulumi/pulumi-terraform-bridge/pull/1355/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pulumi)

:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pulumi)

github-actions[bot]: Diff for [pulumi-random](https://github.com/pulumi-bot/pulumi-random/compare/1f5a761..65d565c) with merge commit 4f79e64fbc6a12406b62dd9482cdd8c798dba424
github-actions[bot]: Diff for [pulumi-azuread](https://github.com/pulumi-bot/pulumi-azuread/compare/409d55c..f804b77) with merge commit 4f79e64fbc6a12406b62dd9482cdd8c798dba424

Patch:
diff --git a/pf/tfbridge/provider_create.go b/pf/tfbridge/provider_create.go
index 6a1117f4..7394cb2c 100644
--- a/pf/tfbridge/provider_create.go
+++ b/pf/tfbridge/provider_create.go
@@ -66,6 +66,16 @@ func (p *provider) CreateWithContext(
 		if err != nil {
 			return "", nil, 0, err
 		}
+
+		if rh.pulumiResourceInfo.TransformOutputs != nil {
+			var err error
+			plannedStatePropertyMap, err = rh.pulumiResourceInfo.TransformOutputs(ctx,
+				plannedStatePropertyMap)
+			if err != nil {
+				return "", nil, 0, err
+			}
+		}
+
 		return "", plannedStatePropertyMap, resource.StatusOK, nil
 	}
 
@@ -104,6 +114,14 @@ func (p *provider) CreateWithContext(
 		return "", nil, 0, err
 	}
 
+	if rh.pulumiResourceInfo.TransformOutputs != nil {
+		var err error
+		createdStateMap, err = rh.pulumiResourceInfo.TransformOutputs(ctx, createdStateMap)
+		if err != nil {
+			return "", nil, 0, err
+		}
+	}
+
 	createdID, err := createdState.ExtractID(&rh)
 	if err != nil {
 		return "", nil, 0, err
diff --git a/pf/tfbridge/provider_read.go b/pf/tfbridge/provider_read.go
index fe99e541..e21031d0 100644
--- a/pf/tfbridge/provider_read.go
+++ b/pf/tfbridge/provider_read.go
@@ -65,6 +65,14 @@ func (p *provider) ReadWithContext(
 		return result, ignoredStatus, err
 	}
 
+	if result.Outputs != nil && rh.pulumiResourceInfo.TransformOutputs != nil {
+		var err error
+		result.Outputs, err = rh.pulumiResourceInfo.TransformOutputs(ctx, result.Outputs)
+		if err != nil {
+			return result, ignoredStatus, err
+		}
+	}
+
 	if result.Outputs != nil {
 		result.Inputs, err = tfbridge.ExtractInputsFromOutputs(
 			oldInputs,
diff --git a/pf/tfbridge/provider_update.go b/pf/tfbridge/provider_update.go
index e86e8efe..9a2e3f1e 100644
--- a/pf/tfbridge/provider_update.go
+++ b/pf/tfbridge/provider_update.go
@@ -77,6 +77,16 @@ func (p *provider) UpdateWithContext(
 		if err != nil {
 			return nil, 0, err
 		}
+
+		if rh.pulumiResourceInfo.TransformOutputs != nil {
+			var err error
+			plannedStatePropertyMap, err = rh.pulumiResourceInfo.TransformOutputs(ctx,
+				plannedStatePropertyMap)
+			if err != nil {
+				return nil, 0, err
+			}
+		}
+
 		return plannedStatePropertyMap, resource.StatusOK, nil
 	}
 
@@ -112,5 +122,13 @@ func (p *provider) UpdateWithContext(
 		return nil, 0, err
 	}
 
+	if rh.pulumiResourceInfo.TransformOutputs != nil {
+		var err error
+		updatedStateMap, err = rh.pulumiResourceInfo.TransformOutputs(ctx, updatedStateMap)
+		if err != nil {
+			return nil, 0, err
+		}
+	}
+
 	return updatedStateMap, resource.StatusOK, nil
 }
diff --git a/pkg/tfbridge/info.go b/pkg/tfbridge/info.go
index f3476c17..686bc505 100644
--- a/pkg/tfbridge/info.go
+++ b/pkg/tfbridge/info.go
@@ -323,6 +323,13 @@ type ResourceInfo struct {
 
 	// An experimental way to augment the Check function in the Pulumi life cycle.
 	PreCheckCallback PreCheckCallback
+
+	// Resource operations such as Create, Read, and Update return the resource outputs to be
+	// recored in Pulumi statefile. TransformOutputs provides the last chance to edit these
+	// outputs before they are stored. In particular, it can be used as a last resort hook to
+	// make corrections in the default translation of the resource state from TF to Pulumi.
+	// Should be used sparingly.
+	TransformOutputs func(ctx context.Context, outputs resource.PropertyMap) (resource.PropertyMap, error)
 }
 
 type PreCheckCallback = func(
diff --git a/pkg/tfbridge/provider.go b/pkg/tfbridge/provider.go
index 60133ef7..ea20a2a9 100644
--- a/pkg/tfbridge/provider.go
+++ b/pkg/tfbridge/provider.go
@@ -776,6 +776,14 @@ func (p *Provider) Create(ctx context.Context, req *pulumirpc.CreateRequest) (*p
 		reasons = append(reasons, errors.Wrapf(err, "converting result for %s", urn).Error())
 	}
 
+	if res.Schema.TransformOutputs != nil {
+		var err error
+		props, err = res.Schema.TransformOutputs(ctx, props)
+		if err != nil {
+			return nil, err
+		}
+	}
+
 	mprops, err := plugin.MarshalProperties(props, plugin.MarshalOptions{
 		Label:        fmt.Sprintf("%s.outs", label),
 		KeepUnknowns: req.GetPreview(),
@@ -788,6 +796,7 @@ func (p *Provider) Create(ctx context.Context, req *pulumirpc.CreateRequest) (*p
 	if len(reasons) != 0 {
 		return nil, initializationError(newstate.ID(), mprops, reasons)
 	}
+
 	return &pulumirpc.CreateResponse{Id: newstate.ID(), Properties: mprops}, nil
 }
 
@@ -852,6 +861,14 @@ func (p *Provider) Read(ctx context.Context, req *pulumirpc.ReadRequest) (*pulum
 			return nil, err
 		}
 
+		if res.Schema.TransformOutputs != nil {
+			var err error
+			props, err = res.Schema.TransformOutputs(ctx, props)
+			if err != nil {
+				return nil, err
+			}
+		}
+
 		mprops, err := plugin.MarshalProperties(props, plugin.MarshalOptions{
 			Label:       label + ".state",
 			KeepSecrets: p.supportsSecrets,
@@ -967,6 +984,15 @@ func (p *Provider) Update(ctx context.Context, req *pulumirpc.UpdateRequest) (*p
 	if err != nil {
 		reasons = append(reasons, errors.Wrapf(err, "converting result for %s", urn).Error())
 	}
+
+	if res.Schema.TransformOutputs != nil {
+		var err error
+		props, err = res.Schema.TransformOutputs(ctx, props)
+		if err != nil {
+			return nil, err
+		}
+	}
+
 	mprops, err := plugin.MarshalProperties(props, plugin.MarshalOptions{
 		Label:        fmt.Sprintf("%s.outs", label),
 		KeepUnknowns: req.GetPreview(),


Test patch:
diff --git a/pf/tests/internal/testprovider/testbridge_resource_testcompres.go b/pf/tests/internal/testprovider/testbridge_resource_testcompres.go
index eb64baaf..d3ea0c46 100644
--- a/pf/tests/internal/testprovider/testbridge_resource_testcompres.go
+++ b/pf/tests/internal/testprovider/testbridge_resource_testcompres.go
@@ -30,6 +30,8 @@ type testCompRes struct{}
 
 var _ resource.Resource = &testCompRes{}
 
+var _ resource.ResourceWithImportState = &testCompRes{}
+
 func newTestCompRes() resource.Resource {
 	return &testCompRes{}
 }
@@ -89,9 +91,14 @@ func (e *testCompRes) Read(ctx context.Context, req resource.ReadRequest, resp *
 }
 
 func (e *testCompRes) Update(ctx context.Context, req resource.UpdateRequest, resp *resource.UpdateResponse) {
-	panic("TODO Update")
+	resp.State.Raw = req.Plan.Raw.Copy()
 }
 
 func (e *testCompRes) Delete(ctx context.Context, req resource.DeleteRequest, resp *resource.DeleteResponse) {
 	panic("TODO Delete")
 }
+
+func (e *testCompRes) ImportState(ctx context.Context, req resource.ImportStateRequest, resp *resource.ImportStateResponse) {
+	resp.State.SetAttribute(ctx, path.Root("id"), "someID")
+	resp.State.SetAttribute(ctx, path.Root("ecdsacurve"), "P384")
+}
diff --git a/pf/tests/provider_transform_outputs_test.go b/pf/tests/provider_transform_outputs_test.go
new file mode 100644
index 00000000..c4d1f1ad
--- /dev/null
+++ b/pf/tests/provider_transform_outputs_test.go
@@ -0,0 +1,143 @@
+// Copyright 2016-2023, Pulumi Corporation.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package tfbridgetests
+
+import (
+	"context"
+	"testing"
+
+	"github.com/pulumi/pulumi-terraform-bridge/pf/tests/internal/testprovider"
+	testutils "github.com/pulumi/pulumi-terraform-bridge/testing/x"
+	"github.com/pulumi/pulumi/sdk/v3/go/common/resource"
+)
+
+func TestTransformOutputs(t *testing.T) {
+	p := testprovider.SyntheticTestBridgeProvider()
+
+	p.Resources["testbridge_testcompres"].TransformOutputs = func(
+		_ context.Context,
+		pm resource.PropertyMap,
+	) (resource.PropertyMap, error) {
+		c := pm.Copy()
+		c["ecdsacurve"] = resource.NewStringProperty("TRANSFORMED")
+		return c, nil
+	}
+	provider := newProviderServer(t, p)
+
+	t.Run("Create preview", func(t *testing.T) {
+		testutils.Replay(t, provider, `
+		{
+		  "method": "/pulumirpc.ResourceProvider/Create",
+		  "request": {
+		    "urn": "urn:pulumi:dev::teststack::testbridge:index/testres:Testcompres::exres",
+		    "properties": {},
+		    "preview": true
+		  },
+		  "response": {
+		    "properties": {
+		      "id": "*",
+		      "ecdsacurve": "TRANSFORMED"
+		    }
+		  }
+		}`)
+	})
+
+	t.Run("Create", func(t *testing.T) {
+		testutils.Replay(t, provider, `
+		{
+		  "method": "/pulumirpc.ResourceProvider/Create",
+		  "request": {
+		    "urn": "urn:pulumi:dev::teststack::testbridge:index/testres:Testcompres::exres",
+		    "properties": {
+                      "ecdsacurve": "P384"
+                    }
+		  },
+		  "response": {
+                    "id": "*",
+		    "properties": {
+		      "id": "*",
+		      "ecdsacurve": "TRANSFORMED"
+		    }
+		  }
+		}`)
+	})
+
+	t.Run("Update preview", func(t *testing.T) {
+		testutils.Replay(t, provider, `
+		{
+		  "method": "/pulumirpc.ResourceProvider/Update",
+		  "request": {
+		    "id": "0",
+		    "urn": "urn:pulumi:dev::teststack::testbridge:index/testres:Testcompres::exres",
+		    "olds": {
+                      "ecdsacurve": "P384"
+		    },
+		    "news": {
+                      "ecdsacurve": "P385"
+		    },
+	            "preview": true
+		  },
+		  "response": {
+		    "properties": {
+		      "id": "*",
+	              "ecdsacurve": "TRANSFORMED"
+		    }
+		  }
+		}`)
+	})
+
+	t.Run("Update", func(t *testing.T) {
+		testutils.Replay(t, provider, `
+		{
+		  "method": "/pulumirpc.ResourceProvider/Update",
+		  "request": {
+		    "id": "0",
+		    "urn": "urn:pulumi:dev::teststack::testbridge:index/testres:Testcompres::exres",
+		    "olds": {
+                      "ecdsacurve": "P384"
+		    },
+		    "news": {
+                      "ecdsacurve": "P385"
+		    }
+		  },
+		  "response": {
+		    "properties": {
+		      "id": "*",
+	              "ecdsacurve": "TRANSFORMED"
+		    }
+		  }
+		}`)
+	})
+
+	t.Run("Read to import", func(t *testing.T) {
+		testutils.Replay(t, provider, `
+		{
+		  "method": "/pulumirpc.ResourceProvider/Read",
+		  "request": {
+		    "id": "0",
+		    "urn": "urn:pulumi:dev::teststack::testbridge:index/testres:Testcompres::exres",
+	            "properties": {}
+		  },
+		  "response": {
+	            "id": "*",
+	            "inputs": "*",
+		    "properties": {
+			"id": "*",
+                        "ecdsacurve": "TRANSFORMED"
+		    }
+		  }
+		}`)
+	})
+}
diff --git a/pkg/tfbridge/provider_test.go b/pkg/tfbridge/provider_test.go
index 0c1698d8..e6674808 100644
--- a/pkg/tfbridge/provider_test.go
+++ b/pkg/tfbridge/provider_test.go
@@ -1627,3 +1627,168 @@ func TestInvoke(t *testing.T) {
 		}`)
 	})
 }
+
+func TestTransformOutputs(t *testing.T) {
+	provider := &Provider{
+		tf:     shimv2.NewProvider(testTFProviderV2),
+		config: shimv2.NewSchemaMap(testTFProviderV2.Schema),
+		resources: map[tokens.Type]Resource{
+			"ExampleResource": {
+				TF:     shimv2.NewResource(testTFProviderV2.ResourcesMap["example_resource"]),
+				TFName: "example_resource",
+				Schema: &ResourceInfo{
+					Tok: "ExampleResource",
+					TransformOutputs: func(
+						ctx context.Context,
+						pm resource.PropertyMap,
+					) (resource.PropertyMap, error) {
+						p := pm.Copy()
+						p["stringPropertyValue"] = resource.NewStringProperty("TRANSFORMED")
+						return p, nil
+					},
+				},
+			},
+		},
+	}
+
+	t.Run("Create preview", func(t *testing.T) {
+		testutils.Replay(t, provider, `
+		{
+		  "method": "/pulumirpc.ResourceProvider/Create",
+		  "request": {
+		    "urn": "urn:pulumi:dev::teststack::ExampleResource::exres",
+		    "properties": {
+		      "__defaults": [],
+		      "stringPropertyValue": "SOME"
+		    },
+		    "preview": true
+		  },
+		  "response": {
+		    "properties": {
+		      "id": "",
+		      "stringPropertyValue": "TRANSFORMED"
+		    }
+		  }
+		}`)
+	})
+
+	t.Run("Create", func(t *testing.T) {
+		testutils.Replay(t, provider, `
+		{
+		  "method": "/pulumirpc.ResourceProvider/Create",
+		  "request": {
+		    "urn": "urn:pulumi:dev::teststack::ExampleResource::exres",
+		    "properties": {
+		      "__defaults": [],
+		      "boolPropertyValue": true
+		    }
+		  },
+		  "response": {
+		    "id": "*",
+		    "properties": {
+		      "id": "*",
+                      "stringPropertyValue": "TRANSFORMED",
+		      "boolPropertyValue": "*",
+		      "__meta": "*",
+		      "objectPropertyValue": "*",
+		      "floatPropertyValue": "*",
+		      "stringPropertyValue": "*",
+		      "arrayPropertyValues": "*",
+		      "nestedResources": "*",
+		      "numberPropertyValue": "*",
+		      "setPropertyValues": "*",
+		      "stringWithBadInterpolation": "*"
+		    }
+		  }
+		}`)
+	})
+
+	t.Run("Update preview", func(t *testing.T) {
+		testutils.Replay(t, provider, `
+		{
+		  "method": "/pulumirpc.ResourceProvider/Update",
+		  "request": {
+		    "id": "0",
+		    "urn": "urn:pulumi:dev::teststack::ExampleResource::exres",
+		    "olds": {
+		      "stringPropertyValue": "OLD"
+		    },
+		    "news": {
+		      "stringPropertyValue": "NEW"
+		    },
+                    "preview": true
+		  },
+		  "response": {
+		    "properties": {
+		      "id": "*",
+                      "stringPropertyValue": "TRANSFORMED",
+		      "__meta": "*",
+		      "arrayPropertyValues": "*",
+		      "nestedResources": "*",
+		      "setPropertyValues": "*"
+		    }
+		  }
+		}`)
+	})
+
+	t.Run("Update", func(t *testing.T) {
+		testutils.Replay(t, provider, `
+		{
+		  "method": "/pulumirpc.ResourceProvider/Update",
+		  "request": {
+		    "id": "0",
+		    "urn": "urn:pulumi:dev::teststack::ExampleResource::exres",
+		    "olds": {
+		      "stringPropertyValue": "OLD"
+		    },
+		    "news": {
+		      "stringPropertyValue": "NEW"
+		    }
+		  },
+		  "response": {
+		    "properties": {
+		      "id": "*",
+		      "stringPropertyValue": "TRANSFORMED",
+		      "boolPropertyValue": "*",
+		      "__meta": "*",
+		      "objectPropertyValue": "*",
+		      "floatPropertyValue": "*",
+		      "arrayPropertyValues": "*",
+		      "nestedResources": "*",
+		      "numberPropertyValue": "*",
+		      "setPropertyValues": "*",
+		      "stringWithBadInterpolation": "*"
+		    }
+		  }
+		}`)
+	})
+
+	t.Run("Read to import", func(t *testing.T) {
+		testutils.Replay(t, provider, `
+		{
+		  "method": "/pulumirpc.ResourceProvider/Read",
+		  "request": {
+		    "id": "0",
+		    "urn": "urn:pulumi:dev::teststack::ExampleResource::exres",
+                    "properties": {}
+		  },
+		  "response": {
+                    "id": "0",
+                    "inputs": "*",
+		    "properties": {
+			"id": "*",
+			"stringPropertyValue": "TRANSFORMED",
+			"boolPropertyValue": "*",
+			"__meta": "*",
+			"objectPropertyValue": "*",
+			"floatPropertyValue": "*",
+			"arrayPropertyValues": "*",
+			"nestedResources": "*",
+			"numberPropertyValue": "*",
+			"setPropertyValues": "*",
+			"stringWithBadInterpolation": "*"
+		    }
+		  }
+		}`)
+	})
+}