Repository: cal-itp/benefits

Issue:
have a way to test Terraform changes more safely
Currently, our different environments are set up as App Service slots. This means infrastructure (a.k.a. Terraform) changes outside of slots apply to _all of them_, meaning there isn't an easy way to test those changes beyond a `plan`.

I propose we use a dedicated Resource Group (maybe also a dedicated subscription?) to create a full `staging` environment, perhaps with some features like the Monitors disabled. We might want to go all the way and make one for `dev`, and get rid of our use of slots. Open to feedback here.

cc @cal-itp/cdt-devsecops 

## Acceptance Criteria

- [x] Changes can be made to Terraform tested without concern that production will be impacted

## Additional context

This would have helped prevent [the 7/21 downtime](https://docs.google.com/document/d/14XFpJonXmrNKDt98g8vML3RimbZha0UJiIeUVclXYGs/edit#).


Discussion:
More support for this, from the official docs:

> notably, you shouldn't use a single Terraform workspace to manage everything that makes up your production or staging environment

https://developer.hashicorp.com/terraform/cloud-docs/recommended-practices/part1#one-workspace-per-environment-per-terraform-configuration

Resolving patch:
diff --git a/terraform/pipeline/workspace.py b/terraform/pipeline/workspace.py
new file mode 100644
--- /dev/null
+++ b/terraform/pipeline/workspace.py
@@ -0,0 +1,46 @@
+"""Used to set the environment-related variables at runtime (rather than build
+time) so that all the necessary pipeline variables are available."""
+
+import os
+import sys
+
+REASON = os.environ["REASON"]
+# the name of the variable that Azure Pipelines uses for the source branch
+# depends on the type of run, so need to check both
+SOURCE = os.environ.get("OTHER_SOURCE") or os.environ["INDIVIDUAL_SOURCE"]
+TARGET = os.environ["TARGET"]
+
+# the branches that correspond to environments
+ENV_BRANCHES = ["dev", "test", "prod"]
+
+if REASON == "PullRequest" and TARGET in ENV_BRANCHES:
+    # it's a pull request against one of the environment branches, so use the
+    # target branch
+    environment = TARGET
+elif REASON in ["IndividualCI", "Manual"] and SOURCE in ENV_BRANCHES:
+    # it's being run on one of the environment branches, so use that
+    environment = SOURCE
+else:
+    # default to running against dev
+    environment = "dev"
+
+# matching logic in ../init.sh
+workspace = "default" if environment == "prod" else environment
+
+service_connection = "Production" if environment == "prod" else "Development"
+
+# just for troubleshooting
+if TARGET is not None:
+    deployment_description = f"from {SOURCE} to {TARGET}"
+else:
+    deployment_description = f"for {SOURCE}"
+print(
+    f"Deploying {deployment_description}",
+    f"as a result of {REASON}",
+    f"using workspace {workspace}," f"and service connection {service_connection}",
+    file=sys.stderr,
+)
+
+# https://learn.microsoft.com/en-us/azure/devops/pipelines/process/set-variables-scripts?view=azure-devops&tabs=bash#about-tasksetvariable
+print(f"##vso[task.setvariable variable=workspace;isOutput=true]{workspace}")
+print(f"##vso[task.setvariable variable=service_connection;isOutput=true]{service_connection}")


Test patch: