Repository: localstack/terraform-local (language: python)

Issue:
bug: when configuring tflocal for remote s3 state, live AWS is used
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

When setting up remote state using dynamoDB and s3, localstack currently behaves as follows:

1. Create a bucket and dynamodb table either using awslocal or tflocal.
3. Write a terraform.backend block for a simple terraform configuration to reference the backend bucket and table.
   
```
terraform {
  backend "s3" {
    bucket = "backend-bucket-iac-terraform"
    key    = "terraform.tfstate"
    region = "us-west-2"

    dynamodb_table = "terraform-state-locking"

    # s3_use_path_style           = true
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    # skip_requesting_account_id  = true
  }
}
```
3. Now run `tflocal init`.  This will FAIL, saying that the bucket was not found.
4. Now create the corresponding backend bucket and dynamodb table *on live AWS* with the same names and regions as you used for localstack. 
5. Without changing your terraform configuration, run `tflocal init` again. IT WILL SUCCEED, and moreover, the live S3 bucket on AWS will contain your remote state (including the fake ARN refs to the created objects on localstack).

This is behavior is somewhat unexpected :-)

I have [a repo containing the demonstration files](https://github.com/torenware/localstash-remote-state-issue) for anyone who'd like to try this out for themselves.

### Expected Behavior

I'd expect `tflocal` to look for the bucket and dynamodb table locally, on the localstack mocked db, and not on live AWS :-) 

### How are you starting LocalStack?

With the `localstack` script

### Steps To Reproduce

#### How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)

    $ localstack start

#### Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

    1. Download my demo repo
    2. Follow the instructions in the repo for creating the bucket and dynamodb table set up in the `remote-state` directory.
    3. `cd ../dyn1`
    4. `tflocal init`
    5. Observe the error returned; tflocal did not find the remote state bucket:
     
```
Initializing the backend...
╷
│ Error: Failed to get existing workspaces: S3 bucket does not exist.
│
│ The referenced S3 bucket must have been previously created. If the S3 bucket
│ was created within the last minute, please wait for a minute or two and try
│ again.
│
│ Error: NoSuchBucket: The specified bucket does not exist
│ 	status code: 404, request id: 7WRR8Q21Y5DTWVAV, host id: Zb0SMz3HOprcP8kDQo9QRMJ2YtrbjFqUW0rtWJIWqqnlykGRS1yaZTmZAfS/5aiGVyK9BLbIaV8=
│
│
│
```
    7. Now create a bucket with the same name and region on *live AWS*.
    8. Run `tflocal init` again, and observe that init works, and you can apply the configuration as well.  The resources will be created on localstash, but the bucket and dynamodb table on *live AWS* will be updated with the remote state.


### Environment

```markdown
- OS:  MacOS Big Sur, using Colima for docker.
- LocalStack: 2.1.0
- awscli v2 is configured for us-west-2 with a valid key and secret.
```


### Anything else?

_No response_


Discussion / hints:
Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our [Slack Community](https://localstack.cloud/contact/) for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our [contributing guidelines](https://github.com/localstack/.github/blob/main/CONTRIBUTING.md) and our [contributing guide](https://docs.localstack.cloud/contributing/).
Hi @torenware,

Could you please try using this terraform code and running the `tflocal init`, you might need to run it with `-reconfigure` option?

```terraform
terraform {
  backend "s3" {
    bucket                      = "mybucket"
    key                         = "terraform.tfstate"
    region                      = "us-east-1"
    dynamodb_table              = "terraform-state-lock"
    endpoint                    = "http://s3.localhost.localstack.cloud:4566"
    iam_endpoint                = "http://localhost:4566"
    sts_endpoint                = "http://localhost:4566"
    dynamodb_endpoint           = "http://localhost:4566"
  }
}

```
This time, it appears that if `mybucket` does not exist in the localstack server, `tflocal init` gets a "bucket does not exist" error.  If however I create the bucket,  `tflocal init` now SUCCEEDS.  I'm testing out `tflocal apply` now; there's an issue with the dynomodb lock table right now I'm looking into. 
OK, if the lock table appears in the correct region, apply also works.

So the terraform.backend block also needs endpoint and *_endpoint arguments?  Is this documented anywhere?


OK, I see [it's documented here.](https://developer.hashicorp.com/terraform/language/settings/backends/s3#credentials-and-shared-configuration).  This is however a work-around; tflocal should not be going back to real-world AWS, yes?
Sorry, I forgot to mention the documentation earlier. 
Yes, this is a workaround and I will move this issue under the https://github.com/localstack/terraform-local

As `tflocal` is a wrapper, you can create your configuration file using the same technique https://developer.hashicorp.com/terraform/language/files/override.


Fix patch:
diff --git a/README.md b/README.md
index 53212d7..d44e4f9 100644
--- a/README.md
+++ b/README.md
@@ -37,6 +37,7 @@ please refer to the man pages of `terraform --help`.
 
 ## Change Log
 
+* v0.10: Add support for storing state files in local S3 backends
 * v0.9: Fix unsupported provider override for emrserverless
 * v0.8: Configure the endpoint for opensearch service
 * v0.7: Add initial support for provider aliases
diff --git a/bin/tflocal b/bin/tflocal
index 9821e62..c1aaa61 100755
--- a/bin/tflocal
+++ b/bin/tflocal
@@ -42,9 +42,32 @@ provider "aws" {
  }
 }
 """
+TF_S3_BACKEND_CONFIG = """
+terraform {
+  backend "s3" {
+    region         = "<region>"
+    bucket         = "<bucket>"
+    key            = "<key>"
+    dynamodb_table = "<dynamodb_table>"
+
+    access_key        = "test"
+    secret_key        = "test"
+    endpoint          = "<s3_endpoint>"
+    iam_endpoint      = "<iam_endpoint>"
+    sts_endpoint      = "<sts_endpoint>"
+    dynamodb_endpoint = "<dynamodb_endpoint>"
+    skip_credentials_validation = true
+    skip_metadata_api_check     = true
+  }
+}
+"""
 PROCESS = None
 
 
+# ---
+# CONFIG GENERATION UTILS
+# ---
+
 def create_provider_config_file(provider_aliases=None):
     provider_aliases = provider_aliases or []
 
@@ -98,6 +121,9 @@ def create_provider_config_file(provider_aliases=None):
     # construct final config file content
     tf_config = "\n".join(provider_configs)
 
+    # create s3 backend config
+    tf_config += generate_s3_backend_config()
+
     # write temporary config file
     providers_file = get_providers_file_path()
     if os.path.exists(providers_file):
@@ -117,26 +143,61 @@ def get_providers_file_path() -> str:
     return os.path.join(base_dir, LS_PROVIDERS_FILE)
 
 
-def get_service_endpoint(service: str) -> str:
-    # allow configuring a custom endpoint via the environment
-    env_name = f"{service.replace('-', '_').upper().strip()}_ENDPOINT"
-    env_endpoint = os.environ.get(env_name, "").strip()
-    if env_endpoint:
-        if "://" not in env_endpoint:
-            env_endpoint = f"http://{env_endpoint}"
-        return env_endpoint
+def determine_provider_aliases() -> list:
+    """Return a list of providers (and aliases) configured in the *.tf files (if any)"""
+    result = []
+    tf_files = parse_tf_files()
+    for _file, obj in tf_files.items():
+        try:
+            providers = ensure_list(obj.get("provider", []))
+            aws_providers = [prov["aws"] for prov in providers if prov.get("aws")]
+            result.extend(aws_providers)
+        except Exception as e:
+            print(f"Warning: Unable to extract providers from {_file}:", e)
+    return result
 
-    # some services need specific hostnames
-    hostname = LOCALSTACK_HOSTNAME
-    if service == "s3":
-        hostname = S3_HOSTNAME
-    elif service == "mwaa":
-        hostname = f"mwaa.{LOCALHOST_HOSTNAME}"
 
-    return f"http://{hostname}:{EDGE_PORT}"
+def generate_s3_backend_config() -> str:
+    """Generate an S3 `backend {..}` block with local endpoints, if configured"""
+    backend_config = None
+    tf_files = parse_tf_files()
+    for obj in tf_files.values():
+        tf_configs = ensure_list(obj.get("terraform", []))
+        for tf_config in tf_configs:
+            backend_config = ensure_list(tf_config.get("backend"))
+            if backend_config:
+                backend_config = backend_config[0]
+                break
+    backend_config = backend_config and backend_config.get("s3")
+    if not backend_config:
+        return ""
+
+    configs = {
+        # note: default values, updated by `backend_config` further below...
+        "bucket": "tf-test-state",
+        "key": "terraform.tfstate",
+        "dynamodb_table": "tf-test-state",
+        "region": get_region(),
+        "s3_endpoint": get_service_endpoint("s3"),
+        "iam_endpoint": get_service_endpoint("iam"),
+        "sts_endpoint": get_service_endpoint("sts"),
+        "dynamodb_endpoint": get_service_endpoint("dynamodb"),
+    }
+    configs.update(backend_config)
+    get_or_create_bucket(configs["bucket"])
+    get_or_create_ddb_table(configs["dynamodb_table"], region=configs["region"])
+    result = TF_S3_BACKEND_CONFIG
+    for key, value in configs.items():
+        result = result.replace(f"<{key}>", value)
+    return result
 
 
+# ---
+# AWS CLIENT UTILS
+# ---
+
 def use_s3_path_style() -> bool:
+    """Whether to use S3 path addressing (depending on the configured S3 endpoint)"""
     regex = r"^[a-z]+://(localhost|[0-9.]+)(:[0-9]+)?$"
     return bool(re.match(regex, get_service_endpoint("s3")))
 
@@ -157,27 +218,74 @@ def get_region() -> str:
     return region or DEFAULT_REGION
 
 
-def to_bytes(obj) -> bytes:
-    return obj.encode("UTF-8") if isinstance(obj, str) else obj
+def get_service_endpoint(service: str) -> str:
+    """Get the service endpoint URL for the given service name"""
+    # allow configuring a custom endpoint via the environment
+    env_name = f"{service.replace('-', '_').upper().strip()}_ENDPOINT"
+    env_endpoint = os.environ.get(env_name, "").strip()
+    if env_endpoint:
+        if "://" not in env_endpoint:
+            env_endpoint = f"http://{env_endpoint}"
+        return env_endpoint
 
+    # some services need specific hostnames
+    hostname = LOCALSTACK_HOSTNAME
+    if service == "s3":
+        hostname = S3_HOSTNAME
+    elif service == "mwaa":
+        hostname = f"mwaa.{LOCALHOST_HOSTNAME}"
 
-def to_str(obj) -> bytes:
-    return obj.decode("UTF-8") if isinstance(obj, bytes) else obj
+    return f"http://{hostname}:{EDGE_PORT}"
 
 
-def determine_provider_aliases() -> list:
-    """Return a list of providers (and aliases) configured in the *.tf files (if any)"""
-    result = []
+def connect_to_service(service: str, region: str = None):
+    import boto3
+    region = region or get_region()
+    return boto3.client(
+        service, endpoint_url=get_service_endpoint(service), region_name=region,
+        aws_access_key_id="test", aws_secret_access_key="test",
+    )
+
+
+def get_or_create_bucket(bucket_name: str):
+    """Get or create a bucket in the current region."""
+    s3_client = connect_to_service("s3")
+    try:
+        return s3_client.head_bucket(Bucket=bucket_name)
+    except Exception:
+        region = s3_client.meta.region_name
+        kwargs = {}
+        if region != "us-east-1":
+            kwargs = {"CreateBucketConfiguration": {"LocationConstraint": region}}
+        return s3_client.create_bucket(Bucket=bucket_name, **kwargs)
+
+
+def get_or_create_ddb_table(table_name: str, region: str = None):
+    """Get or create a DynamoDB table with the given name."""
+    ddb_client = connect_to_service("dynamodb", region=region)
+    try:
+        return ddb_client.describe_table(TableName=table_name)
+    except Exception:
+        return ddb_client.create_table(
+            TableName=table_name, BillingMode="PAY_PER_REQUEST",
+            KeySchema=[{"AttributeName": "LockID", "KeyType": "HASH"}],
+            AttributeDefinitions=[{"AttributeName": "LockID", "AttributeType": "S"}]
+        )
+
+
+# ---
+# TF UTILS
+# ---
+
+def parse_tf_files() -> dict:
+    """Parse the local *.tf files and return a dict of <filename> -> <resource_dict>"""
+    result = {}
     for _file in glob.glob('*.tf'):
         try:
             with open(_file, 'r') as fp:
-                obj = hcl2.load(fp)
-            providers = obj.get("provider", [])
-            providers = [providers] if not isinstance(providers, list) else providers
-            aws_providers = [prov["aws"] for prov in providers if prov.get("aws")]
-            result.extend(aws_providers)
+                result[_file] = hcl2.load(fp)
         except Exception as e:
-            print(f"Warning: Unable to extract providers from {_file}:", e)
+            print(f"Unable to parse '{_file}' as HCL file: {e}")
     return result
 
 
@@ -201,10 +309,30 @@ def run_tf_subprocess(cmd, env):
     sys.exit(PROCESS.returncode)
 
 
+# ---
+# UTIL FUNCTIONS
+# ---
+
 def signal_handler(sig, frame):
     PROCESS.send_signal(sig)
 
 
+def ensure_list(obj) -> list:
+    return obj if isinstance(obj, list) else [obj]
+
+
+def to_bytes(obj) -> bytes:
+    return obj.encode("UTF-8") if isinstance(obj, str) else obj
+
+
+def to_str(obj) -> bytes:
+    return obj.decode("UTF-8") if isinstance(obj, bytes) else obj
+
+
+# ---
+# MAIN ENTRYPOINT
+# ---
+
 def main():
     env = dict(os.environ)
     cmd = [TF_CMD] + sys.argv[1:]
diff --git a/setup.cfg b/setup.cfg
index 7e7bab9..279cddf 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -1,6 +1,6 @@
 [metadata]
 name = terraform-local
-version = 0.9
+version = 0.10
 url = https://github.com/localstack/terraform-local
 author = LocalStack Team
 author_email = info@localstack.cloud


Test patch:
diff --git a/tests/test_apply.py b/tests/test_apply.py
index 6b496f1..aaf6281 100644
--- a/tests/test_apply.py
+++ b/tests/test_apply.py
@@ -64,6 +64,43 @@ def test_provider_aliases(monkeypatch):
     assert any(queue_name2 in queue_url for queue_url in queues2)
 
 
+def test_s3_backend():
+    state_bucket = f"tf-state-{short_uid()}"
+    state_table = f"tf-state-{short_uid()}"
+    bucket_name = f"bucket.{short_uid()}"
+    config = """
+    terraform {
+      backend "s3" {
+        bucket = "%s"
+        key    = "terraform.tfstate"
+        dynamodb_table = "%s"
+        region = "us-east-2"
+      }
+    }
+    resource "aws_s3_bucket" "test-bucket" {
+      bucket = "%s"
+    }
+    """ % (state_bucket, state_table, bucket_name)
+    deploy_tf_script(config)
+
+    # assert that bucket with state file exists
+    s3 = client("s3", region_name="us-east-2")
+    result = s3.list_objects(Bucket=state_bucket)
+    keys = [obj["Key"] for obj in result["Contents"]]
+    assert "terraform.tfstate" in keys
+
+    # assert that DynamoDB table with state file locks exists
+    dynamodb = client("dynamodb", region_name="us-east-2")
+    result = dynamodb.describe_table(TableName=state_table)
+    attrs = result["Table"]["AttributeDefinitions"]
+    assert attrs == [{"AttributeName": "LockID", "AttributeType": "S"}]
+
+    # assert that S3 resource has been created
+    s3 = client("s3")
+    result = s3.head_bucket(Bucket=bucket_name)
+    assert result["ResponseMetadata"]["HTTPStatusCode"] == 200
+
+
 ###
 # UTIL FUNCTIONS
 ###