Repository: gruntwork-io/terragrunt

Issue:
Terragrunt ignores the version constraint of a Terraform module
## Describe the bug

Terragrunt ignores the version constraint of a Terraform module and instead wants to use a previous version of the AWS provider. 

## Steps To Reproduce

The following output shows how the `.terraform.lock.hcl` has been generated and that Terragrunt ignores the version constraint of a Terraform module. 

### Terraform

```hcl
❯ cat versions.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }

  required_version = ">= 1.0"
}

```

```hcl
❯ terraform init -upgrade
Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 5.0"...
- Installing hashicorp/aws v5.66.0...
- Installed hashicorp/aws v5.66.0 (signed by HashiCorp)
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
```

```hcl
❯ cat .terraform.lock.hcl
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/aws" {
  version     = "5.66.0"
  constraints = "~> 5.0"
  hashes = [
    "h1:yGcVdhj9IKbS/b7BSHtgGjCiFnKK+81ImkK/x7UCgEI=",
    "zh:071c908eb18627f4becdaf0a9fe95d7a61f69be365080aba2ef5e24f6314392b",
    "zh:3dea2a474c6ad4be5b508de4e90064ec485e3fbcebb264cb6c4dec660e3ea8b5",
    "zh:56c0b81e3bbf4e9ccb2efb984f8758e2bc563ce179ff3aecc1145df268b046d1",
    "zh:5f34b75a9ef69cad8c79115ecc0697427d7f673143b81a28c3cf8d5decfd7f93",
    "zh:65632bc2c408775ee44cb32a72e7c48376001a9a7b3adbc2c9b4d088a7d58650",
    "zh:6d0550459941dfb39582fadd20bfad8816255a827bfaafb932d51d66030fcdd5",
    "zh:7f1811ef179e507fdcc9776eb8dc3d650339f8b84dd084642cf7314c5ca26745",
    "zh:8a793d816d7ef57e71758fe95bf830cfca70d121df70778b65cc11065ad004fd",
    "zh:8c7cda08adba01b5ae8cc4e5fbf16761451f0fab01327e5f44fc47b7248ba653",
    "zh:96d855f1771342771855c0fb2d47ff6a731e8f2fa5d242b18037c751fd63e6c3",
    "zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425",
    "zh:b2a62669b72c2471820410b58d764102b11c24e326831ddcfae85c7d20795acf",
    "zh:b4a6b251ac24c8f5522581f8d55238d249d0008d36f64475beefc3791f229e1d",
    "zh:ca519fa7ee1cac30439c7e2d311a0ecea6a5dae2d175fe8440f30133688b6272",
    "zh:fbcd54e7d65806b0038fc8a0fbdc717e1284298ff66e22aac39dcc5a22cc99e5",
  ]
}
```

### Terragrunt

```hcl
❯ terragrunt init -upgrade
09:30:59.187 INFO   Terragrunt Cache server is listening on 127.0.0.1:59846
09:30:59.187 INFO   Start Terragrunt Cache server
09:30:59.576 INFO   Downloading Terraform configurations from file:///Users/dhoppe/Documents/customers/siemens/terragrunt/terragrunt-aws-nbm/modules/aws-data into /Users/dhoppe/Documents/customers/siemens/terragrunt/terragrunt-aws-nbm/stacks/dev/us-east-2/aws-data/.terragrunt-cache/1klylyiqYO0KFyf0u1oUkTjrYxw/KrtfoKBczd_PpPJjZfeW7fDSg8s
09:31:03.336 INFO   Caching terraform providers for /Users/dhoppe/Documents/customers/siemens/terragrunt/terragrunt-aws-nbm/stacks/dev/us-east-2/aws-data/.terragrunt-cache/1klylyiqYO0KFyf0u1oUkTjrYxw/KrtfoKBczd_PpPJjZfeW7fDSg8s
09:31:07.772 STDOUT terraform: Initializing the backend...
09:31:09.851 STDOUT terraform: Initializing provider plugins...
09:31:09.851 STDOUT terraform: - Finding hashicorp/aws versions matching "~> 5.0"...
09:31:09.915 STDOUT terraform: - Installing hashicorp/aws v5.66.0...
09:31:10.491 STDOUT terraform: - Installed hashicorp/aws v5.66.0 (unauthenticated)
09:31:10.491 STDOUT terraform: Terraform has been successfully initialized!
09:31:10.491 STDOUT terraform:
09:31:10.491 STDOUT terraform: You may now begin working with Terraform. Try running "terraform plan" to see
09:31:10.491 STDOUT terraform: any changes that are required for your infrastructure. All Terraform commands
09:31:10.491 STDOUT terraform: should now work.
09:31:10.491 STDOUT terraform: If you ever set or change modules or backend configuration for Terraform,
09:31:10.491 STDOUT terraform: rerun this command to reinitialize your working directory. If you forget, other
09:31:10.491 STDOUT terraform: commands will detect it and remind you to do so if necessary.
09:31:10.492 INFO   Shutting down Terragrunt Cache server...
09:31:10.492 INFO   Terragrunt Cache server stopped
```

```hcl
❯ cat .terraform.lock.hcl
# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/aws" {
  version     = "5.66.0"
  constraints = "5.66.0"
  hashes = [
    "h1:yGcVdhj9IKbS/b7BSHtgGjCiFnKK+81ImkK/x7UCgEI=",
    "zh:071c908eb18627f4becdaf0a9fe95d7a61f69be365080aba2ef5e24f6314392b",
    "zh:3dea2a474c6ad4be5b508de4e90064ec485e3fbcebb264cb6c4dec660e3ea8b5",
    "zh:56c0b81e3bbf4e9ccb2efb984f8758e2bc563ce179ff3aecc1145df268b046d1",
    "zh:5f34b75a9ef69cad8c79115ecc0697427d7f673143b81a28c3cf8d5decfd7f93",
    "zh:65632bc2c408775ee44cb32a72e7c48376001a9a7b3adbc2c9b4d088a7d58650",
    "zh:6d0550459941dfb39582fadd20bfad8816255a827bfaafb932d51d66030fcdd5",
    "zh:7f1811ef179e507fdcc9776eb8dc3d650339f8b84dd084642cf7314c5ca26745",
    "zh:8a793d816d7ef57e71758fe95bf830cfca70d121df70778b65cc11065ad004fd",
    "zh:8c7cda08adba01b5ae8cc4e5fbf16761451f0fab01327e5f44fc47b7248ba653",
    "zh:96d855f1771342771855c0fb2d47ff6a731e8f2fa5d242b18037c751fd63e6c3",
    "zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425",
    "zh:b2a62669b72c2471820410b58d764102b11c24e326831ddcfae85c7d20795acf",
    "zh:b4a6b251ac24c8f5522581f8d55238d249d0008d36f64475beefc3791f229e1d",
    "zh:ca519fa7ee1cac30439c7e2d311a0ecea6a5dae2d175fe8440f30133688b6272",
    "zh:fbcd54e7d65806b0038fc8a0fbdc717e1284298ff66e22aac39dcc5a22cc99e5",
  ]
}
```

## Expected behavior

Terragrunt should keep the version constraint of a Terraform module, like `~> 5.0` instead of a static version `5.66.0`. 

## Nice to haves

- [x] Terminal output
- [ ] Screenshots

## Versions

- Terragrunt version: Terragrunt v0.67.4
- OpenTofu/Terraform version: Terraform v1.9.5
- Environment details (Ubuntu 20.04, Windows 10, etc.): Ubuntu 22.04

## Additional context

The current behaviour causes some trouble regarding our GitLab CI pipeline. 

```
terraform: │ Error: Failed to query available provider packages
terraform: │
terraform: │ Could not retrieve the list of available versions for provider
terraform: │ hashicorp/aws: locked provider registry.terraform.io/hashicorp/aws 5.65.0
terraform: │ does not match configured version constraint ~> 5.66; must use terraform
terraform: │ init -upgrade to allow selection of new versions
```

Maybe I am wrong, but I would expect Terragrunt to simply use the last available AWS provider and not bother with the version constraint in the `.terraform.lock.hcl` of Terragrunt and instead follow the requirement of the Terraform module. 

Discussion:
@dhoppe: I have been thinking a bit about my own bug report and I am afraid Terragrunt is working as expected. After all, the `.terraform.lock.hcl` ensures that Terragrunt works with the previously defined versions. 

Even if I use a dedicated repository for each Terraform module and Renovate takes care of the `.terraform.lock.hcl`, this does not create a new version which could then be used by Terragrunt to generate a new `.terraform.lock.hcl` via Renovate based on the latest version of the AWS provider. 🤔 @RafaelWO: I have a similar issue, although my main problem is that I want to understand why Terragrunt "now" (not sure since when) always sets `constraints` to the same value as `version` after running e.g. `terragrunt init`.

![Screenshot from 2024-10-04 12-21-25](https://github.com/user-attachments/assets/bffe11a2-bb4f-4296-be40-313d58d1b33e)

The lock file in the screenshot was generated with an older version of Terragrunt (likely v0.55.5) which basically uses the version specifier from the `required_providers` block of the TF module as `constraints`.

But now (I'm using v0.67.16) Terragrunt always wants to update `constraints` to have the same value as `version`. Is this expected behavior? I.e. should I update all my lock files to have `constraints` == `version`?

---

I'm using the following Terrgrunt flags:
```
TERRAGRUNT_PROVIDER_CACHE=true
TERRAGRUNT_FETCH_DEPENDENCY_OUTPUT_FROM_STATE=true
TERRAGRUNT_USE_PARTIAL_PARSE_CONFIG_CACHE=true
```@gnuletik: We have the exact same issue as @RafaelWO.

Since circa July, when running `terragrunt init` (with or without the `-upgrade` flag), the `constraints` field in `.terraform.lock.hcl` is updated with a fixed version instead of the "sum" of module constraints.

Thie new behavior works when running `terragrunt` locally.
However, this change broke the https://github.com/renovatebot/renovate/ integration that updates terargrunt lock files (because renovate relies on constraints).

Is that behavior intended?

Should I open another issue for this?@JulesClaussen: Short update after digging, this started at this version 0.58.3 https://github.com/gruntwork-io/terragrunt/releases/tag/v0.58.3, when you added the lock file generation in the provider caching.
Based on this comment, https://github.com/gruntwork-io/terragrunt/pull/3108/files#diff-94642adf6319f65c9f28d52d5ab91229ef3265269b04fe827bd843f2f293d362R92 it appears to be by design. What's the reason behind that?@RafaelWO: I noticed that I get the expected behaviour, i.e. version ranges in `constraints`, if I disable the provider cache.

```
TERRAGRUNT_PROVIDER_CACHE=false terragrunt providers lock
```

But then the provider cache is obviously not used :wink: @lev-ok: This is not normal behavior when the Terragrunt cache provider is enabled.

If `.terraform.lock.hcl` is missing, there are two ways to create it:
1. Run `terraform/tofu init` in this case all providers from the remote registry will be downloaded, even if they are present in the cache dir.
2. Generate it itself, this is what Terragrunt provider cache does, but it can't calculate `constraints`, therefore it sets the value of the current version.@RafaelWO: But I observe that Terragrunt replaces the value in `constraints` even if the `.terraform.lock.hcl` exists.@JulesClaussen: > 2\. Generate it itself, this is what Terragrunt provider cache does, but it can't calculate `constraints`, therefore it sets the value of the current version.

What do you mean it can't calculate constraints? Is there no way to reproduce what is done without the provider cache, within it? As RafaelWO mentioned, even though the .terraform.lock.hcl exists, it gets updated to **remove** the constraints. That's quite an unexpected behaviour.@yhakbar: Definitely worth addressing, and if anyone from the community is interested in digging into provider cache server code, I would welcome taking a look at it. It's a large undertaking to maintain this functionality, as we're working around gaps in what OpenTofu/Terraform can't do natively.

We'll get to this eventually if nobody from the community addresses it.

Solution patch:
diff --git a/tf/getproviders/lock.go b/tf/getproviders/lock.go
index 50ab803b..9302f9fe 100644
--- a/tf/getproviders/lock.go
+++ b/tf/getproviders/lock.go
@@ -16,6 +16,7 @@ import (
 	"github.com/gruntwork-io/terragrunt/internal/errors"
 	"github.com/gruntwork-io/terragrunt/tf"
 	"github.com/gruntwork-io/terragrunt/util"
+	"github.com/hashicorp/go-version"
 	"github.com/hashicorp/hcl/v2"
 	"github.com/hashicorp/hcl/v2/hclsyntax"
 	"github.com/hashicorp/hcl/v2/hclwrite"
@@ -90,8 +91,32 @@ func updateProviderBlock(ctx context.Context, providerBlock *hclwrite.Block, pro
 
 	providerBlock.Body().SetAttributeValue("version", cty.StringVal(provider.Version()))
 
-	// Constraints can contain multiple constraint expressions, including comparison operators, but in the Terragrunt Provider Cache use case, we assume that the required_providers are pinned to a specific version to detect the required version without terraform init, so we can simply specify the constraints attribute as the same as the version. This may differ from what terraform generates, but we expect that it doesn't matter in practice.
-	providerBlock.Body().SetAttributeValue("constraints", cty.StringVal(provider.Version()))
+	// If version constraints exist in current lock file and match the new version, we keep them unchanged.
+	// Otherwise, we specify the constraints attribute the same as the version.
+	currentConstraintsAttr := providerBlock.Body().GetAttribute("constraints")
+
+	shouldUpdateConstraints := false
+
+	if currentConstraintsAttr != nil {
+		currentConstraints, err := version.NewConstraint(strings.ReplaceAll(string(currentConstraintsAttr.Expr().BuildTokens(nil).Bytes()), `"`, ""))
+		// If current version constraints are malformed, we should update it.
+		if err != nil {
+			shouldUpdateConstraints = true
+		}
+
+		newVersion, _ := version.NewVersion(provider.Version())
+		// If current version constrains do not match the new provider version, we should update it.
+		if !currentConstraints.Check(newVersion) {
+			shouldUpdateConstraints = true
+		}
+	} else {
+		// If there is no constraints attribute, we should update it.
+		shouldUpdateConstraints = true
+	}
+
+	if shouldUpdateConstraints {
+		providerBlock.Body().SetAttributeValue("constraints", cty.StringVal(provider.Version()))
+	}
 
 	h1Hash, err := PackageHashV1(provider.PackageDir())
 	if err != nil {



Test patch:
diff --git a/tf/getproviders/lock_test.go b/tf/getproviders/lock_test.go
index dee705dc..372f59fc 100644
--- a/tf/getproviders/lock_test.go
+++ b/tf/getproviders/lock_test.go
@@ -117,6 +117,51 @@ provider "registry.terraform.io/hashicorp/azurerm" {
   ]
 }
 
+provider "registry.terraform.io/hashicorp/template" {
+  version     = "2.2.0"
+  constraints = "2.2.0"
+  hashes = [
+    "h1:kvJsWhTmFya0WW8jAfY40fDtYhWQ6mOwPQC2ncDNjZs=",
+    "zh:02d170f0a0f453155686baf35c10b5a7a230ef20ca49f6e26de1c2691ac70a59",
+    "zh:d88ec10849d5a1d9d1db458847bbc62049f0282a2139e5176d645b75a0346992",
+  ]
+}
+`,
+		},
+		{
+			providers: []getproviders.Provider{},
+			initialLockfile: `
+provider "registry.terraform.io/hashicorp/aws" {
+  version     = "5.36.0"
+  constraints = ">= 5.36.0"
+  hashes = [
+    "h1:SHOEBOHEif46z7Bb86YZ5evCtAeK5A4gtHdT8RU5OhA=",
+    "zh:7c810fb11d8b3ded0cb554a27c27a9d002cc644a7a57c29cae01eeea890f0398",
+    "zh:a3366f6b57b0f4b8bf8a5fecf42a834652709a97dd6db1b499c4ab186e33a41f",
+  ]
+}
+			
+provider "registry.terraform.io/hashicorp/template" {
+  version     = "2.1.0"
+  constraints = "<= 2.1.0" 
+  hashes = [
+    "h1:vxE/PD8SWl6Lmh5zRvIW1Y559xfUyuV2T/VeQLXi7f0=",
+    "zh:6fc271665ac28c3fee773b9dc2b8066280ba35b7e9a14a6148194a240c43f42a",
+    "zh:c19f719c9f7ce6d7449fe9c020100faed0705303c7f95beeef81dfd1e4a2004b",
+  ]
+}			
+`,
+			expectedLockfile: `
+provider "registry.terraform.io/hashicorp/aws" {
+  version     = "5.37.0"
+  constraints = ">= 5.36.0"
+  hashes = [
+    "h1:SHOEBOHEif46z7Bb86YZ5evCtAeK5A4gtHdT8RU5OhA=",
+    "zh:7c810fb11d8b3ded0cb554a27c27a9d002cc644a7a57c29cae01eeea890f0398",
+    "zh:a3366f6b57b0f4b8bf8a5fecf42a834652709a97dd6db1b499c4ab186e33a41f",
+  ]
+}
+
 provider "registry.terraform.io/hashicorp/template" {
   version     = "2.2.0"
   constraints = "2.2.0"
@@ -147,6 +192,11 @@ provider "registry.terraform.io/hashicorp/template" {
 					mockProviderUpdateLock(t, ctrl, "registry.terraform.io/hashicorp/aws", "5.36.0"),
 					mockProviderUpdateLock(t, ctrl, "registry.terraform.io/hashicorp/template", "2.2.0"),
 				}
+			case 2:
+				tc.providers = []getproviders.Provider{
+					mockProviderUpdateLock(t, ctrl, "registry.terraform.io/hashicorp/aws", "5.37.0"),
+					mockProviderUpdateLock(t, ctrl, "registry.terraform.io/hashicorp/template", "2.2.0"),
+				}
 			}
 
 			workingDir := t.TempDir()