Repository: runatlantis/atlantis (language: go)

Issue:
Pin terraform versions in tests
<!--- Please keep this note for the community --->

### Community Note

- Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
- Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
- If you are interested in working on this issue or have submitted a pull request, please leave a comment.

<!--- Thank you for keeping this note for the community --->

---

- [X] I'd be willing to implement this feature ([contributing guide](https://github.com/runatlantis/atlantis/blob/main/CONTRIBUTING.md))

**Describe the user story**

@nitrocode recommended in https://github.com/runatlantis/atlantis/pull/4462#discussion_r1573148934 that we pin terraform versions in tests. I didn't want to block that particular fix because unit tests were failing, but it seems like a good idea that I'm happy to work on.

**Describe the solution you'd like**

Terraform providers are pinned in tests such that external updates don't cause tests to start failing.

**Describe the drawbacks of your solution**

We may fall behind new features, but this is only in unit tests so the risk is low.

**Describe alternatives you've considered**

We could simply do nothing and live with the occasional test failure. I personally feel strongly about working to make tests idempotent and independent of changes in the "outside world", so I think pinning is the right strategy.



Discussion / hints:

Fix patch:
diff --git a/scripts/pin_ci_terraform_providers.sh b/scripts/pin_ci_terraform_providers.sh
new file mode 100755
index 0000000000..8de2dfab6c
--- /dev/null
+++ b/scripts/pin_ci_terraform_providers.sh
@@ -0,0 +1,74 @@
+#!/bin/bash
+
+# Script to pin terraform providers in e2e tests
+
+RANDOM_PROVIDER_VERSION="3.6.1"
+NULL_PROVIDER_VERSION="3.2.2"
+
+TEST_REPOS_DIR="server/controllers/events/testdata/test-repos"
+
+for file in $(find $TEST_REPOS_DIR -name '*.tf')
+do
+    basename=$(basename $file)
+    if [[ "$basename" == "versions.tf" ]]
+    then
+        continue
+    fi
+    if [[ "$basename" != "main.tf" ]]
+    then
+        echo "Found unexpected file: $file"
+        exit 1
+    fi
+    has_null_provider=false
+    has_random_provider=false
+
+    version_file="$(dirname $file)/versions.tf"
+    for resource in $(cat $file | grep '^resource' | awk '{print $2}' | tr -d '"')
+    do
+        if [[ "$resource" == "null_resource" ]]
+        then
+            has_null_provider=true
+        elif [[ "$resource" == "random_id" ]]
+        then
+            has_random_provider=true
+        else
+            echo "Unknown resource $resource in $file"
+            exit 1
+        fi
+    done
+    if ! $has_null_provider && ! $has_random_provider
+    then
+        echo "No providers needed for $file"
+        continue
+    fi
+    echo "Adding $version_file for $file"
+    rm -f $version_file
+    if $has_null_provider
+    then
+        echo 'provider "null" {}' >> $version_file
+    fi
+    if $has_random_provider
+    then
+        echo 'provider "random" {}' >> $version_file
+    fi
+    echo "terraform {" >> $version_file
+    echo "  required_providers {" >> $version_file
+
+    if $has_random_provider
+    then
+        echo "    random = {" >> $version_file
+        echo '      source  = "hashicorp/random"' >> $version_file
+        echo "      version = \"= $RANDOM_PROVIDER_VERSION\"" >> $version_file
+        echo "    }" >> $version_file
+    fi
+    if $has_null_provider
+    then
+        echo "    null = {" >> $version_file
+        echo '      source  = "hashicorp/null"' >> $version_file
+        echo "      version = \"= $NULL_PROVIDER_VERSION\"" >> $version_file
+        echo "    }" >> $version_file
+    fi
+    echo "  }" >> $version_file
+    echo "}" >> $version_file
+
+done


Test patch:
diff --git a/server/controllers/events/testdata/test-repos/automerge/dir1/versions.tf b/server/controllers/events/testdata/test-repos/automerge/dir1/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/automerge/dir1/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/automerge/dir2/versions.tf b/server/controllers/events/testdata/test-repos/automerge/dir2/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/automerge/dir2/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/import-multiple-project/dir1/versions.tf b/server/controllers/events/testdata/test-repos/import-multiple-project/dir1/versions.tf
new file mode 100644
index 0000000000..66c59fa2b4
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/import-multiple-project/dir1/versions.tf
@@ -0,0 +1,9 @@
+provider "random" {}
+terraform {
+  required_providers {
+    random = {
+      source  = "hashicorp/random"
+      version = "= 3.6.1"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/import-multiple-project/dir2/versions.tf b/server/controllers/events/testdata/test-repos/import-multiple-project/dir2/versions.tf
new file mode 100644
index 0000000000..66c59fa2b4
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/import-multiple-project/dir2/versions.tf
@@ -0,0 +1,9 @@
+provider "random" {}
+terraform {
+  required_providers {
+    random = {
+      source  = "hashicorp/random"
+      version = "= 3.6.1"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/import-single-project-var/versions.tf b/server/controllers/events/testdata/test-repos/import-single-project-var/versions.tf
new file mode 100644
index 0000000000..66c59fa2b4
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/import-single-project-var/versions.tf
@@ -0,0 +1,9 @@
+provider "random" {}
+terraform {
+  required_providers {
+    random = {
+      source  = "hashicorp/random"
+      version = "= 3.6.1"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/import-single-project/versions.tf b/server/controllers/events/testdata/test-repos/import-single-project/versions.tf
new file mode 100644
index 0000000000..66c59fa2b4
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/import-single-project/versions.tf
@@ -0,0 +1,9 @@
+provider "random" {}
+terraform {
+  required_providers {
+    random = {
+      source  = "hashicorp/random"
+      version = "= 3.6.1"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/import-workspace/dir1/versions.tf b/server/controllers/events/testdata/test-repos/import-workspace/dir1/versions.tf
new file mode 100644
index 0000000000..66c59fa2b4
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/import-workspace/dir1/versions.tf
@@ -0,0 +1,9 @@
+provider "random" {}
+terraform {
+  required_providers {
+    random = {
+      source  = "hashicorp/random"
+      version = "= 3.6.1"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/modules-yaml/modules/null/versions.tf b/server/controllers/events/testdata/test-repos/modules-yaml/modules/null/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/modules-yaml/modules/null/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/modules/modules/null/versions.tf b/server/controllers/events/testdata/test-repos/modules/modules/null/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/modules/modules/null/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/policy-checks-apply-reqs/versions.tf b/server/controllers/events/testdata/test-repos/policy-checks-apply-reqs/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/policy-checks-apply-reqs/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/policy-checks-clear-approval/versions.tf b/server/controllers/events/testdata/test-repos/policy-checks-clear-approval/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/policy-checks-clear-approval/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/policy-checks-custom-run-steps/versions.tf b/server/controllers/events/testdata/test-repos/policy-checks-custom-run-steps/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/policy-checks-custom-run-steps/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/policy-checks-diff-owner/versions.tf b/server/controllers/events/testdata/test-repos/policy-checks-diff-owner/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/policy-checks-diff-owner/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/policy-checks-disabled-previous-match/versions.tf b/server/controllers/events/testdata/test-repos/policy-checks-disabled-previous-match/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/policy-checks-disabled-previous-match/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/policy-checks-disabled-repo-server-side/versions.tf b/server/controllers/events/testdata/test-repos/policy-checks-disabled-repo-server-side/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/policy-checks-disabled-repo-server-side/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/policy-checks-disabled-repo/versions.tf b/server/controllers/events/testdata/test-repos/policy-checks-disabled-repo/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/policy-checks-disabled-repo/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/policy-checks-enabled-repo-server-side/versions.tf b/server/controllers/events/testdata/test-repos/policy-checks-enabled-repo-server-side/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/policy-checks-enabled-repo-server-side/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/policy-checks-enabled-repo/versions.tf b/server/controllers/events/testdata/test-repos/policy-checks-enabled-repo/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/policy-checks-enabled-repo/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/policy-checks-extra-args/versions.tf b/server/controllers/events/testdata/test-repos/policy-checks-extra-args/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/policy-checks-extra-args/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/policy-checks-multi-projects/dir1/versions.tf b/server/controllers/events/testdata/test-repos/policy-checks-multi-projects/dir1/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/policy-checks-multi-projects/dir1/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/policy-checks-multi-projects/dir2/versions.tf b/server/controllers/events/testdata/test-repos/policy-checks-multi-projects/dir2/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/policy-checks-multi-projects/dir2/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/policy-checks/versions.tf b/server/controllers/events/testdata/test-repos/policy-checks/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/policy-checks/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/repo-config-file/infrastructure/production/versions.tf b/server/controllers/events/testdata/test-repos/repo-config-file/infrastructure/production/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/repo-config-file/infrastructure/production/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/repo-config-file/infrastructure/staging/versions.tf b/server/controllers/events/testdata/test-repos/repo-config-file/infrastructure/staging/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/repo-config-file/infrastructure/staging/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/server-side-cfg/versions.tf b/server/controllers/events/testdata/test-repos/server-side-cfg/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/server-side-cfg/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/simple-with-lockfile/versions.tf b/server/controllers/events/testdata/test-repos/simple-with-lockfile/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/simple-with-lockfile/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/simple-yaml/versions.tf b/server/controllers/events/testdata/test-repos/simple-yaml/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/simple-yaml/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/simple/versions.tf b/server/controllers/events/testdata/test-repos/simple/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/simple/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/state-rm-multiple-project/dir1/versions.tf b/server/controllers/events/testdata/test-repos/state-rm-multiple-project/dir1/versions.tf
new file mode 100644
index 0000000000..66c59fa2b4
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/state-rm-multiple-project/dir1/versions.tf
@@ -0,0 +1,9 @@
+provider "random" {}
+terraform {
+  required_providers {
+    random = {
+      source  = "hashicorp/random"
+      version = "= 3.6.1"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/state-rm-multiple-project/dir2/versions.tf b/server/controllers/events/testdata/test-repos/state-rm-multiple-project/dir2/versions.tf
new file mode 100644
index 0000000000..66c59fa2b4
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/state-rm-multiple-project/dir2/versions.tf
@@ -0,0 +1,9 @@
+provider "random" {}
+terraform {
+  required_providers {
+    random = {
+      source  = "hashicorp/random"
+      version = "= 3.6.1"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/state-rm-single-project/versions.tf b/server/controllers/events/testdata/test-repos/state-rm-single-project/versions.tf
new file mode 100644
index 0000000000..66c59fa2b4
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/state-rm-single-project/versions.tf
@@ -0,0 +1,9 @@
+provider "random" {}
+terraform {
+  required_providers {
+    random = {
+      source  = "hashicorp/random"
+      version = "= 3.6.1"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/state-rm-workspace/dir1/versions.tf b/server/controllers/events/testdata/test-repos/state-rm-workspace/dir1/versions.tf
new file mode 100644
index 0000000000..66c59fa2b4
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/state-rm-workspace/dir1/versions.tf
@@ -0,0 +1,9 @@
+provider "random" {}
+terraform {
+  required_providers {
+    random = {
+      source  = "hashicorp/random"
+      version = "= 3.6.1"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/tfvars-yaml-no-autoplan/versions.tf b/server/controllers/events/testdata/test-repos/tfvars-yaml-no-autoplan/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/tfvars-yaml-no-autoplan/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/tfvars-yaml/versions.tf b/server/controllers/events/testdata/test-repos/tfvars-yaml/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/tfvars-yaml/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/workspace-parallel-yaml/production/versions.tf b/server/controllers/events/testdata/test-repos/workspace-parallel-yaml/production/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/workspace-parallel-yaml/production/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}
diff --git a/server/controllers/events/testdata/test-repos/workspace-parallel-yaml/staging/versions.tf b/server/controllers/events/testdata/test-repos/workspace-parallel-yaml/staging/versions.tf
new file mode 100644
index 0000000000..003111ee80
--- /dev/null
+++ b/server/controllers/events/testdata/test-repos/workspace-parallel-yaml/staging/versions.tf
@@ -0,0 +1,9 @@
+provider "null" {}
+terraform {
+  required_providers {
+    null = {
+      source  = "hashicorp/null"
+      version = "= 3.2.2"
+    }
+  }
+}