Repository: crossplane-contrib/provider-terraform (go)

Issue:
Enable terraform provider plugin cache by default
<!--
Thank you for helping to improve Crossplane!

Please be sure to search for open issues before raising a new one. We use issues
for bug reports and feature requests. Please find us at https://slack.crossplane.io
for questions, support, and discussion.
-->

### What problem are you facing?
<!--
Please tell us a little about your use case - it's okay if it's hypothetical!
Leading with this context helps frame the feature request so we can ensure we
implement it sensibly.
--->

Currently provider dependencies are getting downloaded into each Workspace directory by default.

There is https://developer.hashicorp.com/terraform/cli/config/config-file#provider-plugin-cache mechanism and it already can be enabled with

```
  env:
  - name: TF_PLUGIN_CACHE_DIR
    value: /tf
```

in `ControllerConfig` ( thanks @bobh66 for sharing the configuration ) 

### How could Crossplane help solve your problem?

It makes sense to configure plugin cache by default so every Workspace will benefit from the shared cache. 


Pull request description:
Enable terraform provider plugin cache
<!--
Thank you for helping to improve Crossplane!

Please read through https://git.io/fj2m9 if this is your first time opening a
Crossplane pull request. Find us in https://slack.crossplane.io/messages/dev if
you need any help contributing.
-->

### Description of your changes

Implement control over https://developer.hashicorp.com/terraform/cli/config/config-file#provider-plugin-cache and enable it by default.

`ProviderConfig.Spec` is extended with the optional `pluginCache` field which is true by default.

Fixes #122 

I have:

- [x] Read and followed Crossplane's [contribution process].
- [x] Run `make reviewable test` to ensure this PR is ready for review.

### How has this code been tested

Creating a couple of Workspaces with aws provider dependency.

Directory sizes within the controller pod with `providerCache: false`:

```
du -sh /tf/*
237.0M	/tf/28f8bf14-8e51-4577-b2f8-8059f80794c4
237.0M	/tf/4669c743-43f6-468d-9563-bce2c2fb59fb
```

It is visible that aws provider binary is fetched to both of workplaces

Directory sizes within the controller pod with `providerCache: true`:

```
 du -sh /tf/*
72.0K	/tf/4669c743-43f6-468d-9563-bce2c2fb59fb
68.0K	/tf/c4ed7b69-21d8-42a0-8910-80a3c39424ff
72.0K	/tf/f390d499-661c-463c-be8c-831363560a66
237.0M	/tf/plugin-cache
```

It is visible that only `/tf/plugin-cache` contains the actual provider binary and the Workspace directories are becoming very lightweight. 

The workspaces are fine and operational
```
k get workspaces
NAME                      READY   SYNCED   AGE
iamrole-terraform-test3   True    True     11m
iamrole-terraform-test2   True    True     9m53s
```
[contribution process]: https://git.io/fj2m9


Patch:
diff --git a/README.md b/README.md
index 78bb43b..8d53e78 100644
--- a/README.md
+++ b/README.md
@@ -255,6 +255,25 @@ module "relative-path-iam" {
 }
 ```
 
+## Provider Plugin Cache(enabled by default)
+
+[Provider Plugin
+Cache](https://developer.hashicorp.com/terraform/cli/config/config-file#provider-plugin-cache)
+is enabled by default to speed up reconciliation.
+
+In case you need to disable it, set optional `pluginCache` to `false` in
+`ProviderConfig`:
+
+```console
+apiVersion: tf.crossplane.io/v1alpha1
+kind: ProviderConfig
+metadata:
+  name: default
+spec:
+  pluginCache: false
+...
+```
+
 ## Known limitations:
 
 * You must either use remote state or ensure the provider container's `/tf`
diff --git a/apis/v1alpha1/types.go b/apis/v1alpha1/types.go
index 15d6779..8f59786 100644
--- a/apis/v1alpha1/types.go
+++ b/apis/v1alpha1/types.go
@@ -32,6 +32,12 @@ type ProviderConfigSpec struct {
 	// automatically inject Terraform provider configuration blocks.
 	// +optional
 	Configuration *string `json:"configuration,omitempty"`
+
+	// PluginCache enables terraform provider plugin caching mechanism
+	// https://developer.hashicorp.com/terraform/cli/config/config-file#provider-plugin-cache
+	// +optional
+	// +kubebuilder:default=true
+	PluginCache *bool `json:"pluginCache,omitempty"`
 }
 
 // ProviderCredentials required to authenticate.
diff --git a/apis/v1alpha1/zz_generated.deepcopy.go b/apis/v1alpha1/zz_generated.deepcopy.go
index b18111d..aa25f27 100644
--- a/apis/v1alpha1/zz_generated.deepcopy.go
+++ b/apis/v1alpha1/zz_generated.deepcopy.go
@@ -114,6 +114,11 @@ func (in *ProviderConfigSpec) DeepCopyInto(out *ProviderConfigSpec) {
 		*out = new(string)
 		**out = **in
 	}
+	if in.PluginCache != nil {
+		in, out := &in.PluginCache, &out.PluginCache
+		*out = new(bool)
+		**out = **in
+	}
 }
 
 // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ProviderConfigSpec.
diff --git a/internal/controller/workspace/workspace.go b/internal/controller/workspace/workspace.go
index 59aa64b..95a75ca 100644
--- a/internal/controller/workspace/workspace.go
+++ b/internal/controller/workspace/workspace.go
@@ -51,23 +51,25 @@ const (
 	errGetPC        = "cannot get ProviderConfig"
 	errGetCreds     = "cannot get credentials"
 
-	errMkdir           = "cannot make Terraform configuration directory"
-	errRemoteModule    = "cannot get remote Terraform module"
-	errSetGitCredDir   = "cannot set GIT_CRED_DIR environment variable"
-	errWriteCreds      = "cannot write Terraform credentials"
-	errWriteGitCreds   = "cannot write .git-credentials to /tmp dir"
-	errWriteConfig     = "cannot write Terraform configuration " + tfConfig
-	errWriteMain       = "cannot write Terraform configuration " + tfMain
-	errInit            = "cannot initialize Terraform configuration"
-	errWorkspace       = "cannot select Terraform workspace"
-	errResources       = "cannot list Terraform resources"
-	errDiff            = "cannot diff (i.e. plan) Terraform configuration"
-	errOutputs         = "cannot list Terraform outputs"
-	errOptions         = "cannot determine Terraform options"
-	errApply           = "cannot apply Terraform configuration"
-	errDestroy         = "cannot destroy Terraform configuration"
-	errVarFile         = "cannot get tfvars"
-	errDeleteWorkspace = "cannot delete Terraform workspace"
+	errMkdir             = "cannot make Terraform configuration directory"
+	errMkPluginCacheDir  = "cannot make Terraform plugin cache directory"
+	errRemoteModule      = "cannot get remote Terraform module"
+	errSetGitCredDir     = "cannot set GIT_CRED_DIR environment variable"
+	errSetPluginCacheDir = "cannot set TF_PLUGIN_CACHE_DIR environment variable"
+	errWriteCreds        = "cannot write Terraform credentials"
+	errWriteGitCreds     = "cannot write .git-credentials to /tmp dir"
+	errWriteConfig       = "cannot write Terraform configuration " + tfConfig
+	errWriteMain         = "cannot write Terraform configuration " + tfMain
+	errInit              = "cannot initialize Terraform configuration"
+	errWorkspace         = "cannot select Terraform workspace"
+	errResources         = "cannot list Terraform resources"
+	errDiff              = "cannot diff (i.e. plan) Terraform configuration"
+	errOutputs           = "cannot list Terraform outputs"
+	errOptions           = "cannot determine Terraform options"
+	errApply             = "cannot apply Terraform configuration"
+	errDestroy           = "cannot destroy Terraform configuration"
+	errVarFile           = "cannot get tfvars"
+	errDeleteWorkspace   = "cannot delete Terraform workspace"
 
 	gitCredentialsFilename = ".git-credentials"
 )
@@ -243,10 +245,25 @@ func (c *connector) Connect(ctx context.Context, mg resource.Managed) (managed.E
 	tf := c.terraform(dir)
 	o := make([]terraform.InitOption, 0, len(cr.Spec.ForProvider.InitArgs))
 	o = append(o, terraform.WithInitArgs(cr.Spec.ForProvider.InitArgs))
+	// NOTE(ytsarev): user tf provider cache mechanism to speed up
+	// reconciliation, see https://developer.hashicorp.com/terraform/cli/config/config-file#provider-plugin-cache
+	if pc.Spec.PluginCache == nil {
+		pc.Spec.PluginCache = new(bool)
+		*pc.Spec.PluginCache = true
+	}
+	if *pc.Spec.PluginCache {
+		pluginCacheDir := filepath.Join(tfDir, "plugin-cache")
+		if err := c.fs.MkdirAll(pluginCacheDir, 0700); err != nil {
+			return nil, errors.Wrap(err, errMkPluginCacheDir)
+		}
+		if err := os.Setenv("TF_PLUGIN_CACHE_DIR", pluginCacheDir); err != nil {
+			return nil, errors.Wrap(err, errSetPluginCacheDir)
+		}
+		defer os.Unsetenv("TF_PLUGIN_CACHE_DIR") // nolint: errcheck
+	}
 	if err := tf.Init(ctx, o...); err != nil {
 		return nil, errors.Wrap(err, errInit)
 	}
-	// TODO(ytsarev): cache .terraform in /tmp to speed up `terraform init` on next reconcile
 
 	return &external{tf: tf, kube: c.kube}, errors.Wrap(tf.Workspace(ctx, meta.GetExternalName(cr)), errWorkspace)
 }
diff --git a/package/crds/tf.crossplane.io_providerconfigs.yaml b/package/crds/tf.crossplane.io_providerconfigs.yaml
index a5f3aaa..69669fa 100644
--- a/package/crds/tf.crossplane.io_providerconfigs.yaml
+++ b/package/crds/tf.crossplane.io_providerconfigs.yaml
@@ -114,6 +114,11 @@ spec:
                   - source
                   type: object
                 type: array
+              pluginCache:
+                default: true
+                description: PluginCache enables terraform provider plugin caching
+                  mechanism https://developer.hashicorp.com/terraform/cli/config/config-file#provider-plugin-cache
+                type: boolean
             required:
             - credentials
             type: object


Test patch:
diff --git a/internal/controller/workspace/workspace_test.go b/internal/controller/workspace/workspace_test.go
index 821d6c4..472253d 100644
--- a/internal/controller/workspace/workspace_test.go
+++ b/internal/controller/workspace/workspace_test.go
@@ -391,6 +391,51 @@ func TestConnect(t *testing.T) {
 			},
 			want: errors.Wrap(errBoom, errWriteGitCreds),
 		},
+		"PluginCacheMkdirError": {
+			reason: "We should return any error encountered while creating the plugin cache directory",
+			fields: fields{
+				kube: &test.MockClient{
+					MockGet: test.NewMockGetFn(nil, func(obj client.Object) error {
+						if pc, ok := obj.(*v1alpha1.ProviderConfig); ok {
+							pc.Spec.Credentials = []v1alpha1.ProviderCredentials{{
+								Filename: ".git-credentials",
+								Source:   xpv1.CredentialsSourceNone,
+							}}
+							pc.Spec.PluginCache = new(bool)
+							*pc.Spec.PluginCache = true
+						}
+						return nil
+					}),
+				},
+				usage: resource.TrackerFn(func(_ context.Context, _ resource.Managed) error { return nil }),
+				fs: afero.Afero{
+					Fs: &ErrFs{
+						Fs:   afero.NewMemMapFs(),
+						errs: map[string]error{filepath.Join(tfDir, "plugin-cache"): errBoom},
+					},
+				},
+				terraform: func(_ string) tfclient {
+					return &MockTf{
+						MockInit: func(ctx context.Context, o ...terraform.InitOption) error { return nil },
+					}
+				},
+			},
+			args: args{
+				mg: &v1alpha1.Workspace{
+					ObjectMeta: metav1.ObjectMeta{UID: uid},
+					Spec: v1alpha1.WorkspaceSpec{
+						ResourceSpec: xpv1.ResourceSpec{
+							ProviderConfigReference: &xpv1.Reference{},
+						},
+						ForProvider: v1alpha1.WorkspaceParameters{
+							Module: "github.com/crossplane/rocks",
+							Source: v1alpha1.ModuleSourceInline,
+						},
+					},
+				},
+			},
+			want: errors.Wrap(errBoom, errMkPluginCacheDir),
+		},
 		"WriteConfigError": {
 			reason: "We should return any error encountered while writing our crossplane-provider-config.tf file",
 			fields: fields{