Repository: gruntwork-io/terragrunt (language: go)

Issue:
Init concurrency/parallelism issues
I'm trying to track down some issues we've been having lately with what i believe is parallelism.  We updated our environment to use a shared tf plugin cache path to cut down on directory size with all the provider binaries:
```
TF_PLUGIN_CACHE_DIR="${HOME}/.terraform-plugin-cache"
```

Since then, our directory size is much smaller, but we're running into a lot of issues with sporrattic init issues:

```
$ terragrunt run-all init
<...>

#### module 1: (error installing)
│ Error: Required plugins are not installed
│ 
│ The installed provider plugins are not consistent with the packages
│ selected in the dependency lock file:
│   - registry.terraform.io/hashicorp/aws: the cached package for registry.terraform.io/hashicorp/aws 4.61.0 (in .terraform/providers) does not match any of the checksums recorded in the dependency lock file
│ 
│ Terraform uses external plugins to integrate with a variety of different
│ infrastructure services. To download the plugins required for this
│ configuration, run:
│   terraform init
╵

#### module 2: (using previously installed)
 Initializing the backend...
 Initializing modules...

 Initializing provider plugins...
 - Reusing previous version of hashicorp/aws from the dependency lock file
 - Using previously-installed hashicorp/aws v4.61.0

 Terraform has been successfully initialized!

 You may now begin working with Terraform. Try running "terraform plan" to see
 any changes that are required for your infrastructure. All Terraform commands
 should now work.

 If you ever set or change modules or backend configuration for Terraform,
 rerun this command to reinitialize your working directory. If you forget, other
 commands will detect it and remind you to do so if necessary.


#### module 3: (installing seemingly from scratch)
 Initializing the backend...

 Initializing provider plugins...
 - Reusing previous version of hashicorp/aws from the dependency lock file
 - Installing hashicorp/aws v4.61.0...
 - Installed hashicorp/aws v4.61.0 (signed by HashiCorp)

 Terraform has made some changes to the provider dependency selections recorded
 in the .terraform.lock.hcl file. Review those changes and commit them to your
 version control system if they represent changes you intended to make.

 Terraform has been successfully initialized!

 You may now begin working with Terraform. Try running "terraform plan" to see
 any changes that are required for your infrastructure. All Terraform commands
 should now work.

 If you ever set or change modules or backend configuration for Terraform,
 rerun this command to reinitialize your working directory. If you forget, other
 commands will detect it and remind you to do so if necessary.
```

Setting paralleism to 4 seems to stop this from happening.  Is this a known limitation with parallelism and shared plugin cache?  I just can't figure out why my aws provider keeps on having checksum failures.


Discussion / hints:
Hi,
as mentioned in [the documentation](https://developer.hashicorp.com/terraform/cli/config/config-file#provider-plugin-cache)

```
Note: The plugin cache directory is not guaranteed to be concurrency safe. The provider installer's behavior in environments with multiple terraform init calls is undefined.
```

I would suggest running `terragrunt run-all init` with `--terragrunt-parallelism=1`

If you're using Terraform >=1.4.0, you may specifically be running into problems related to changes made in the caching mechanism.

https://github.com/hashicorp/terraform/issues/32205
@bushong1, How can I reproduce the issue locally? Could you please provide a configuration that produces this result? 

Fix patch:
diff --git a/cli/cli_app.go b/cli/cli_app.go
index 9c846cbbfd..04c66c4bda 100644
--- a/cli/cli_app.go
+++ b/cli/cli_app.go
@@ -15,7 +15,6 @@ import (
 	"github.com/gruntwork-io/gruntwork-cli/collections"
 	"github.com/hashicorp/go-multierror"
 	"github.com/mattn/go-zglob"
-	"github.com/sirupsen/logrus"
 	"github.com/urfave/cli"
 
 	"github.com/gruntwork-io/terragrunt/aws_helper"
@@ -845,7 +844,7 @@ func runTerragruntWithConfig(originalTerragruntOptions *options.TerragruntOption
 			// case, we are using the user's working dir here, rather than just looking at the parent dir of the
 			// terragrunt.hcl. However, the default value for the user's working dir, set in options.go, IS just the
 			// parent dir of terragrunt.hcl, so these will likely always be the same.
-			lockFileError = copyLockFile(terragruntOptions.WorkingDir, originalTerragruntOptions.WorkingDir, terragruntOptions.Logger)
+			lockFileError = util.CopyLockFile(terragruntOptions.WorkingDir, originalTerragruntOptions.WorkingDir, terragruntOptions.Logger)
 		}
 
 		return multierror.Append(runTerraformError, lockFileError).ErrorOrNil()
@@ -903,20 +902,6 @@ func shouldCopyLockFile(args []string) bool {
 	return false
 }
 
-// Terraform 0.14 now generates a lock file when you run `terraform init`.
-// If any such file exists, this function will copy the lock file to the destination folder
-func copyLockFile(sourceFolder string, destinationFolder string, logger *logrus.Entry) error {
-	sourceLockFilePath := util.JoinPath(sourceFolder, util.TerraformLockFile)
-	destinationLockFilePath := util.JoinPath(destinationFolder, util.TerraformLockFile)
-
-	if util.FileExists(sourceLockFilePath) {
-		logger.Debugf("Copying lock file from %s to %s", sourceLockFilePath, destinationFolder)
-		return util.CopyFile(sourceLockFilePath, destinationLockFilePath)
-	}
-
-	return nil
-}
-
 // Run the given action function surrounded by hooks. That is, run the before hooks first, then, if there were no
 // errors, run the action, and finally, run the after hooks. Return any errors hit from the hooks or action.
 func runActionWithHooks(description string, terragruntOptions *options.TerragruntOptions, terragruntConfig *config.TerragruntConfig, action func() error) error {
diff --git a/config/dependency.go b/config/dependency.go
index c4d7e737ef..f4c9ff0d96 100644
--- a/config/dependency.go
+++ b/config/dependency.go
@@ -654,6 +654,7 @@ func getTerragruntOutputJsonFromInitFolder(terragruntOptions *options.Terragrunt
 // To do this, this function will:
 // - Create a temporary folder
 // - Generate the backend.tf file with the backend configuration from the remote_state block
+// - Copy the provider lock file, if there is one in the dependency's working directory
 // - Run terraform init and terraform output
 // - Clean up folder once json file is generated
 // NOTE: terragruntOptions should be in the context of the targetConfig already.
@@ -713,6 +714,12 @@ func getTerragruntOutputJsonFromRemoteState(
 	}
 	terragruntOptions.Logger.Debugf("Generated remote state configuration in working dir %s", tempWorkDir)
 
+	// Check for a provider lock file and copy it to the working dir if it exists.
+	terragruntDir := filepath.Dir(terragruntOptions.TerragruntConfigPath)
+	if err := util.CopyLockFile(terragruntDir, tempWorkDir, terragruntOptions.Logger); err != nil {
+		return nil, err
+	}
+
 	// The working directory is now set up to interact with the state, so pull it down to get the json output.
 
 	// First run init to setup the backend configuration so that we can run output.
diff --git a/util/file.go b/util/file.go
index bd1ddf9a62..ea793a23b3 100644
--- a/util/file.go
+++ b/util/file.go
@@ -14,6 +14,7 @@ import (
 	"github.com/gruntwork-io/terragrunt/errors"
 	"github.com/mattn/go-zglob"
 	homedir "github.com/mitchellh/go-homedir"
+	"github.com/sirupsen/logrus"
 )
 
 const TerraformLockFile = ".terraform.lock.hcl"
@@ -511,3 +512,17 @@ type PathIsNotFile struct {
 func (err PathIsNotFile) Error() string {
 	return fmt.Sprintf("%s is not a file", err.path)
 }
+
+// Terraform 0.14 now generates a lock file when you run `terraform init`.
+// If any such file exists, this function will copy the lock file to the destination folder
+func CopyLockFile(sourceFolder string, destinationFolder string, logger *logrus.Entry) error {
+	sourceLockFilePath := JoinPath(sourceFolder, TerraformLockFile)
+	destinationLockFilePath := JoinPath(destinationFolder, TerraformLockFile)
+
+	if FileExists(sourceLockFilePath) {
+		logger.Debugf("Copying lock file from %s to %s", sourceLockFilePath, destinationFolder)
+		return CopyFile(sourceLockFilePath, destinationLockFilePath)
+	}
+
+	return nil
+}


Test patch: