Repository: localstack/terraform-local (language: python)

Issue:
Override aliased providers
# Summary
Currently `tflocal` is only overriding the default AWS provider, but it is common to have more than one AWS provider defined per terraform state, using [aliased providers](https://developer.hashicorp.com/terraform/language/providers/configuration#alias-multiple-provider-configurations). This is most commonly used when mixing resources in different AWS regions in the same terraform state. It would be great if tflocal could also override these aliased providers.

# Current behaviour
Given the following providers:
```
provider "aws" {
  region = eu-west-2
}

provider "aws" {
  alias  = "us_east_1"
  region = "us-east-1"
}
```
And no valid (real) AWS credentials configured, this is the output when running terraform against localstack using `tflocal`:
```
╷
│ Error: configuring Terraform AWS Provider: error validating provider credentials: retrieving caller identity from STS: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 99a8111f-87b0-49e2-8794-fb08b80bdaac, api error InvalidClientTokenId: The security token included in the request is invalid.
│ 
│   with provider["registry.terraform.io/hashicorp/aws"].us_east_1,
│   on aliased-providers.tf line 1, in provider "aws":
│    1: provider "aws" {
│ 
╵
```
 This is of course because the aliased provider was not overriden and therefore the real AWS APIs were used instead of localstack.

# Proposed implementation
This is where it gets tricky: Terraform enforces aliased providers uniqueness so we can't simply use the same approach that `tflocal` currently has of creating a new file with the overridden (default) provider. One solution could be to actually modify the aliased providers defined to point to localstack, making use of https://pypi.org/project/python-hcl2/ for that. Such an implementation would also require resetting the providers to the previous state upon any errors or at the end of the execution - no small change!

# Workarounds
One possible workaround is to isolate the aliased providers in a separate file (i.e. aliased-providers.tf) and have a new file with the overridden providers (i.e. aliased-providers-override.tf.disabled) that terraform won't normally pick up. Then it's a matter of switching around the file extensions of said files before executing `tflocal` and restore the initial file names after the `tflocal` execution. Example using a shell script wrapper:
```
#!/usr/bin/env bash
set -e

disable_aliased_providers_override() {
  [[ -f aliased-providers-override.tf ]] && mv aliased-providers-override.tf aliased-providers-override.tf.disabled;
  [[ -f aliased-providers.tf.disabled ]] && mv aliased-providers.tf.disabled aliased-providers.tf;
}

trap "disable_aliased_providers_override" ERR
mv aliased-providers-override.tf.disabled aliased-providers-override.tf
mv aliased-providers.tf aliased-providers.tf.disabled
tflocal <terraform command goes here>
mv aliased-providers-override.tf aliased-providers-override.tf.disabled
mv aliased-providers.tf.disabled aliased-providers.tf
```
And this is what an overridden aliased provider would look like:
```
provider "aws" {
  alias                       = "us_east_1"
  access_key                  = "test"
  secret_key                  = "test"
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  region                      = "us-east-1"
  endpoints {
    acm        = "http://localhost:4566"
    cloudfront = "http://localhost:4566"
    <any other AWS services that will be using this provider...>
  }
}
```


Discussion / hints:
Thanks for reporting @ricoli , this is a great proposal, and we should definitely look into supporting this use case. 👍 

I'm wondering if we can find an even easier/user-friendly way to support this - for example there exist some HCL parsers in Python out there (e.g., [this](https://github.com/virtuald/pyhcl), or [this](https://github.com/amplify-education/python-hcl2) one), which may come in handy here.. 🤔 

Would you have the capacity to help spike out a first implementation? We'll see if we can allocate some time on our end to work on this, but definitely happy to help shepherd a contribution if you're up for it..! 🙌 

Fix patch:
diff --git a/Makefile b/Makefile
index 65dfcd0..d250c20 100644
--- a/Makefile
+++ b/Makefile
@@ -1,6 +1,7 @@
 VENV_DIR ?= .venv
 VENV_RUN = . $(VENV_DIR)/bin/activate
 PIP_CMD ?= pip
+TEST_PATH ?= tests
 
 usage:        ## Show this help
 	@fgrep -h "##" $(MAKEFILE_LIST) | fgrep -v fgrep | sed -e 's/\\$$//' | sed -e 's/##//'
@@ -14,7 +15,7 @@ lint:         ## Run code linter
 	$(VENV_RUN); flake8 --ignore=E501,W503 bin/tflocal tests
 
 test:         ## Run unit/integration tests
-	$(VENV_RUN); pytest $(PYTEST_ARGS) -sv tests
+	$(VENV_RUN); pytest $(PYTEST_ARGS) -sv $(TEST_PATH)
 
 publish:      ## Publish the library to the central PyPi repository
 	# build and upload archive
diff --git a/bin/tflocal b/bin/tflocal
index 96c5b83..9a324c0 100755
--- a/bin/tflocal
+++ b/bin/tflocal
@@ -12,6 +12,7 @@ your TF config.
 import os
 import re
 import sys
+import glob
 import subprocess
 
 PARENT_FOLDER = os.path.realpath(os.path.join(os.path.dirname(__file__), '..'))
@@ -19,6 +20,7 @@ if os.path.isdir(os.path.join(PARENT_FOLDER, '.venv')):
     sys.path.insert(0, PARENT_FOLDER)
 
 from localstack_client import config  # noqa: E402
+import hcl2  # noqa: E402
 
 DEFAULT_REGION = "us-east-1"
 LOCALHOST_HOSTNAME = "localhost.localstack.cloud"
@@ -43,7 +45,9 @@ provider "aws" {
 PROCESS = None
 
 
-def create_provider_config_file():
+def create_provider_config_file(provider_aliases=None):
+    provider_aliases = provider_aliases or []
+
     # maps services to be replaced with alternative names
     service_replaces = {
         "apigatewaymanagementapi": "",
@@ -71,14 +75,28 @@ def create_provider_config_file():
             pass
     services = sorted(services)
 
-    # create config
-    endpoints = "\n".join([f'{s} = "{get_service_endpoint(s)}"' for s in services])
-    tf_config = TF_PROVIDER_CONFIG.replace("<endpoints>", endpoints)
-    additional_configs = []
-    if use_s3_path_style():
-        additional_configs += [" s3_use_path_style = true"]
-    additional_configs += [f' region = "{get_region()}"']
-    tf_config = tf_config.replace("<configs>", "\n".join(additional_configs))
+    # add default (non-aliased) provider, if not defined yet
+    default_provider = [p for p in provider_aliases if not p.get("alias")]
+    if not default_provider:
+        provider_aliases.append({"region": get_region()})
+
+    # create provider configs
+    provider_configs = []
+    for provider in provider_aliases:
+        endpoints = "\n".join([f'{s} = "{get_service_endpoint(s)}"' for s in services])
+        provider_config = TF_PROVIDER_CONFIG.replace("<endpoints>", endpoints)
+        additional_configs = []
+        if use_s3_path_style():
+            additional_configs += [" s3_use_path_style = true"]
+        if provider.get("alias"):
+            additional_configs += [f' alias = "{provider["alias"]}"']
+        region = provider.get("region") or get_region()
+        additional_configs += [f' region = "{region}"']
+        provider_config = provider_config.replace("<configs>", "\n".join(additional_configs))
+        provider_configs.append(provider_config)
+
+    # construct final config file content
+    tf_config = "\n".join(provider_configs)
 
     # write temporary config file
     providers_file = get_providers_file_path()
@@ -91,6 +109,7 @@ def create_provider_config_file():
 
 
 def get_providers_file_path() -> str:
+    """Determine the path under which the providers override file should be stored"""
     chdir = [arg for arg in sys.argv if arg.startswith("-chdir=")]
     base_dir = "."
     if chdir:
@@ -146,6 +165,22 @@ def to_str(obj) -> bytes:
     return obj.decode("UTF-8") if isinstance(obj, bytes) else obj
 
 
+def determine_provider_aliases() -> list:
+    """Return a list of providers (and aliases) configured in the *.tf files (if any)"""
+    result = []
+    for _file in glob.glob('*.tf'):
+        try:
+            with open(_file, 'r') as fp:
+                obj = hcl2.load(fp)
+            providers = obj.get("provider", [])
+            providers = [providers] if not isinstance(providers, list) else providers
+            aws_providers = [prov["aws"] for prov in providers if prov.get("aws")]
+            result.extend(aws_providers)
+        except Exception as e:
+            print(f"Warning: Unable to extract providers from {_file}:", e)
+    return result
+
+
 def run_tf_exec(cmd, env):
     """Run terraform using os.exec - can be useful as it does not require any I/O
         handling for stdin/out/err. Does *not* allow us to perform any cleanup logic."""
@@ -175,7 +210,8 @@ def main():
     cmd = [TF_CMD] + sys.argv[1:]
 
     # create TF provider config file
-    config_file = create_provider_config_file()
+    providers = determine_provider_aliases()
+    config_file = create_provider_config_file(providers)
 
     # call terraform command
     try:
diff --git a/setup.cfg b/setup.cfg
index 1f71401..8319880 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -1,6 +1,6 @@
 [metadata]
 name = terraform-local
-version = 0.6
+version = 0.7
 url = https://github.com/localstack/terraform-local
 author = LocalStack Team
 author_email = info@localstack.cloud
@@ -26,6 +26,7 @@ packages = find:
 
 install_requires =
     localstack-client
+    python-hcl2
 
 [options.extras_require]
 test =


Test patch:
diff --git a/tests/test_apply.py b/tests/test_apply.py
index 3d6f867..6b496f1 100644
--- a/tests/test_apply.py
+++ b/tests/test_apply.py
@@ -35,6 +35,35 @@ def test_use_s3_path_style(monkeypatch):
     assert use_s3_path_style()  # noqa
 
 
+def test_provider_aliases(monkeypatch):
+    queue_name1 = f"q{short_uid()}"
+    queue_name2 = f"q{short_uid()}"
+    config = """
+    provider "aws" {
+      region = "eu-west-1"
+    }
+    provider "aws" {
+      alias  = "us_east_2"
+      region = "us-east-2"
+    }
+    resource "aws_sqs_queue" "queue1" {
+      name = "%s"
+    }
+    resource "aws_sqs_queue" "queue2" {
+      name = "%s"
+      provider = aws.us_east_2
+    }
+    """ % (queue_name1, queue_name2)
+    deploy_tf_script(config)
+
+    sqs1 = client("sqs", region_name="eu-west-1")
+    sqs2 = client("sqs", region_name="us-east-2")
+    queues1 = [q for q in sqs1.list_queues().get("QueueUrls", [])]
+    queues2 = [q for q in sqs2.list_queues().get("QueueUrls", [])]
+    assert any(queue_name1 in queue_url for queue_url in queues1)
+    assert any(queue_name2 in queue_url for queue_url in queues2)
+
+
 ###
 # UTIL FUNCTIONS
 ###
@@ -54,12 +83,13 @@ def short_uid() -> str:
     return str(uuid.uuid4())[0:8]
 
 
-def client(service: str):
+def client(service: str, **kwargs):
     return boto3.client(
         service,
         aws_access_key_id="test",
         aws_secret_access_key="test",
-        endpoint_url=LOCALSTACK_ENDPOINT
+        endpoint_url=LOCALSTACK_ENDPOINT,
+        **kwargs,
     )