Repository: tenable/terrascan (language: python)

Issue:
Add support for private terraform repos
* terrascan version 1.4.0
* Operating System: MacOS

### Description

A terrascan user posted on [Accurics Community](https://community.accurics.com/t/using-private-terraform-cloud-modules-from-gitlab-ci/106) that when using terraform modules stored on Terraform Cloud, terrascan fails at downloading the private module.

While terraform will make use of the token stored in `~/.terraformrc` as [documented here](https://www.terraform.io/docs/cli/config/config-file.html#credentials-1), terrascan doesn't seem to know of this...yet.

### What I Did
I cloned the aws tf module over to a private tf.io repo for testing, along with a basic `main.tf` that references it. Scanning that project dies as seen below:

```
$ terrascan scan . 
2021-03-26T16:25:10.465-0700	error	downloader/module-download.go:113	error while fetching available modules for module: app.terraform.io/widgetswest/vpc/aws, at registry: app.terraform.io
2021-03-26T16:25:10.465-0700	error	commons/load-dir.go:100	failed to download remote module "app.terraform.io/widgetswest/vpc/aws". error: 'error looking up module versions: 401 Unauthorized'
2021-03-26T16:25:10.465-0700	error	commons/load-dir.go:121	failed to build unified config. errors: <nil>: Failed to read module directory; Module directory  does not exist or cannot be read.

2021-03-26T16:25:10.465-0700	error	cli/run.go:107	scan run failed{error 26 0  failed to build terraform allResourcesConfig}
```



Discussion / hints:
Looks like [terraform cloud auth is done via bearer token](https://www.terraform.io/docs/cloud/api/index.html#authentication), so just need to see if we can add support for that. Think I see how to do so, will tinker over the weekend.
@jlk you mean `private terraform repos` ?
Is there any progress on this?
hey @MrMickS  yep - a little longer than my "over the weekend" comment, sorry. :) 

Wasn't quite as easy as I first thought - we can't just replace the http client, so I had to figure out some of terraform's data structures, plus I've got a few other things going on.

I've got something working POC late last week - adding in ability to read that from `.terraformrc` and doing testing, should have a PR in today or tomorrow.
> @jlk you mean `private terraform repos` ?

doh - thanks @dev-gaur 
That sounds great. Thanks very much.

Fix patch:
diff --git a/docs/usage.md b/docs/usage.md
index 929badb9d..fd3b73dbf 100644
--- a/docs/usage.md
+++ b/docs/usage.md
@@ -139,6 +139,9 @@ $ terrascan scan -t aws -r git -u git@github.com:accurics/KaiMonkey.git//terrafo
 
 The URLs for the remote should follow similar naming as the source argument for modules in Terraform. More details [here](https://www.terraform.io/docs/modules/sources.html).
 
+#### Private terraform module repositories
+When scanning terraform code, terrascan will check for the presence of `~/.terraformrc`. If the file is found, and it contains credential information to authenticate to a private terraform module registry, terrascan will attempt to use the credentials when authenticating to the private repository. For more details on the format of this file, please see Terraform's [config file documentation](https://www.terraform.io/docs/cli/config/config-file.html).
+
 #### Helm
 
 Helm chart can be scanned by specifying "helm" on the -i flag as follows:
diff --git a/go.mod b/go.mod
index e0ec479f9..68a419bd9 100644
--- a/go.mod
+++ b/go.mod
@@ -15,8 +15,10 @@ require (
 	github.com/hashicorp/go-getter v1.5.1
 	github.com/hashicorp/go-retryablehttp v0.6.6
 	github.com/hashicorp/go-version v1.2.1
+	github.com/hashicorp/hcl v1.0.0
 	github.com/hashicorp/hcl/v2 v2.8.2
 	github.com/hashicorp/terraform v0.14.4
+	github.com/hashicorp/terraform-svchost v0.0.0-20200729002733-f050f53b9734
 	github.com/iancoleman/strcase v0.1.3
 	github.com/itchyny/gojq v0.12.1
 	github.com/mattn/go-isatty v0.0.12
@@ -32,7 +34,7 @@ require (
 	github.com/zclconf/go-cty v1.7.1
 	go.uber.org/zap v1.16.0
 	golang.org/x/mod v0.4.2 // indirect
-	golang.org/x/sys v0.0.0-20210320140829-1e4c9ba3b0c4
+	golang.org/x/sys v0.0.0-20210415045647-66c3f260301c
 	gopkg.in/src-d/go-git.v4 v4.13.1
 	gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
 	helm.sh/helm/v3 v3.4.0
diff --git a/go.sum b/go.sum
index 96e79f0fd..3fa0151ee 100644
--- a/go.sum
+++ b/go.sum
@@ -1143,8 +1143,10 @@ golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210113181707-4bcb84eeeb78/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20210320140829-1e4c9ba3b0c4 h1:EZ2mChiOa8udjfp6rRmswTbtZN/QzUQp4ptM4rnjHvc=
-golang.org/x/sys v0.0.0-20210320140829-1e4c9ba3b0c4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210324051608-47abb6519492 h1:Paq34FxTluEPvVyayQqMPgHm+vTOrIifmcYxFBx9TLg=
+golang.org/x/sys v0.0.0-20210324051608-47abb6519492/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210415045647-66c3f260301c h1:6L+uOeS3OQt/f4eFHXZcTxeZrGCuz+CLElgEBjbcTA4=
+golang.org/x/sys v0.0.0-20210415045647-66c3f260301c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
diff --git a/pkg/downloader/module-download.go b/pkg/downloader/module-download.go
index a689a2e57..e5a81ee96 100644
--- a/pkg/downloader/module-download.go
+++ b/pkg/downloader/module-download.go
@@ -18,10 +18,17 @@ package downloader
 
 import (
 	"fmt"
+	"io/ioutil"
 	"os"
 	"path/filepath"
 
+	"github.com/accurics/terrascan/pkg/utils"
 	"github.com/hashicorp/go-version"
+	"github.com/hashicorp/hcl"
+	svchost "github.com/hashicorp/terraform-svchost"
+	"github.com/hashicorp/terraform-svchost/auth"
+	"github.com/hashicorp/terraform-svchost/disco"
+	"github.com/hashicorp/terraform/command/cliconfig"
 	hclConfigs "github.com/hashicorp/terraform/configs"
 	"github.com/hashicorp/terraform/registry"
 	"github.com/hashicorp/terraform/registry/regsrc"
@@ -43,7 +50,71 @@ func newRemoteModuleInstaller() *remoteModuleInstaller {
 func newTerraformRegistryClient() terraformRegistryClient {
 	// get terraform registry client.
 	// terraform registry client provides methods for querying the terraform module registry
-	return registry.NewClient(nil, nil)
+	// if terraformrc file is found, attempt to load credentials from it
+	homedir := utils.GetHomeDir()
+	if len(homedir) == 0 {
+		return registry.NewClient(nil, nil)
+	}
+	// TODO: Need to add support for Windows %APPDIR%/terraform.rc
+	terraformrc := filepath.Join(homedir, ".terraformrc")
+
+	if _, err := os.Stat(terraformrc); os.IsNotExist(err) {
+		return registry.NewClient(nil, nil)
+	}
+	zap.S().Debugf("Found terraform rc file at %s, attempting to parse", terraformrc)
+
+	return NewAuthenticatedRegistryClient(terraformrc)
+}
+
+// NewAuthenticatedRegistryClient parses the contents of a terraformrc file and builds an authenticated
+// registry client using the credentials found in the rcfile
+func NewAuthenticatedRegistryClient(rcFile string) terraformRegistryClient {
+	b, err := ioutil.ReadFile(rcFile)
+	if err != nil {
+		zap.S().Infof("Error reading %s: %s", rcFile, err)
+		return registry.NewClient(nil, nil)
+	}
+
+	services, err := buildDiscoServices(b)
+	if err != nil {
+		zap.S().Infof("Error building terraform credentials %s: %s", rcFile, err)
+		return registry.NewClient(nil, nil)
+	}
+	return registry.NewClient(services, nil)
+}
+
+// buildDiscoServices builds terraform services struct for later authentication to a terraform registry
+func buildDiscoServices(b []byte) (*disco.Disco, error) {
+	obj, err := hcl.Parse(string(b))
+	if err != nil {
+		return nil, err
+	}
+
+	result := &cliconfig.Config{}
+	if err := hcl.DecodeObject(&result, obj); err != nil {
+		return nil, err
+	}
+
+	hostCredentialMap := convertCredentialMapToHostMap(result.Credentials)
+	if hostCredentialMap == nil {
+		return nil, fmt.Errorf("error converting credential map to host map")
+	}
+	credSource := auth.StaticCredentialsSource(hostCredentialMap)
+	services := disco.NewWithCredentialsSource(credSource)
+	return services, nil
+}
+
+// convertCredentialMapToHostMap converts map key to pass to terraform auth builder
+func convertCredentialMapToHostMap(credentialMap map[string]map[string]interface{}) map[svchost.Hostname]map[string]interface{} {
+	hostMap := make(map[svchost.Hostname]map[string]interface{})
+	if credentialMap == nil {
+		return nil
+	}
+
+	for k := range credentialMap {
+		hostMap[svchost.Hostname(k)] = credentialMap[k]
+	}
+	return hostMap
 }
 
 // DownloadModule retrieves the package referenced in the given address
@@ -110,7 +181,7 @@ func (r *remoteModuleInstaller) DownloadRemoteModule(requiredVersion hclConfigs.
 		if registry.IsModuleNotFound(err) {
 			zap.S().Errorf("module: %s, not be found at registry: %s", module.String(), module.Host().Display())
 		} else {
-			zap.S().Errorf("error while fetching available modules for module: %s, at registry: %s", module.String(), module.Host().Display())
+			zap.S().Errorf("error while fetching available modules for module: %s, at registry: %s. Error: %s", module.String(), module.Host().Display(), err.Error())
 		}
 		return "", err
 	}


Test patch:
diff --git a/pkg/downloader/module-download_test.go b/pkg/downloader/module-download_test.go
index 5e0062448..2acff22a8 100644
--- a/pkg/downloader/module-download_test.go
+++ b/pkg/downloader/module-download_test.go
@@ -542,3 +542,101 @@ func TestGetVersionToDownload(t *testing.T) {
 		})
 	}
 }
+
+func TestAuthenticatedRegistryClient(t *testing.T) {
+	testDirPath := filepath.Join("testdata", "run-test")
+
+	tests := []struct {
+		name     string
+		filename string
+	}{
+		{
+			name:     "valid terraformrc file",
+			filename: "terraformrc",
+		},
+		{
+			name:     "invalid terraformrc file",
+			filename: "badterraformrc",
+		},
+		{
+			name:     "invalid terraformrc file",
+			filename: "nonexistantfile",
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			testRCFilePath := filepath.Join(testDirPath, tt.filename)
+			got := NewAuthenticatedRegistryClient(testRCFilePath)
+			if got == nil {
+				t.Errorf("NewAuthenticatedRegistryClient(%s) = nil, expected terraformRegistryClient", testRCFilePath)
+			}
+		})
+	}
+}
+
+func TestBuildDiscoServices(t *testing.T) {
+	testDirPath := "testdata"
+
+	tests := []struct {
+		name     string
+		filename string
+		wantErr  bool
+	}{
+		{
+			name:     "valid terraformrc file to parse",
+			filename: "terraformrc",
+			wantErr:  false,
+		},
+		{
+			name:     "invalid terraformrc file to parse",
+			filename: "badterraformrc",
+			wantErr:  true,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			testRCFilePath := filepath.Join(testDirPath, tt.filename)
+			b, err := ioutil.ReadFile(testRCFilePath)
+			if err != nil {
+				t.Errorf("Error reading %s: %s", testRCFilePath, err)
+				return
+			}
+
+			_, err = buildDiscoServices(b)
+			if (err != nil) != tt.wantErr {
+				t.Errorf("buildDiscoServices() error: %v, wantErr %v", err, tt.wantErr)
+				return
+			}
+		})
+	}
+}
+
+func TestConvertCredentialMapToHostMap(t *testing.T) {
+	tests := []struct {
+		name          string
+		credentialMap map[string]map[string]interface{}
+		wantNilResult bool
+	}{
+		{
+			name:          "nil credentialMap",
+			credentialMap: nil,
+			wantNilResult: true,
+		},
+		{
+			name:          "empty credentialMap",
+			credentialMap: make(map[string]map[string]interface{}),
+			wantNilResult: false,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			hostMap := convertCredentialMapToHostMap(tt.credentialMap)
+			if (hostMap == nil) != tt.wantNilResult {
+				t.Errorf("convertCredentialMapToHostMap() error: got nil result %v, wantNilResult %v", hostMap == nil, tt.wantNilResult)
+				return
+			}
+		})
+	}
+
+}
diff --git a/pkg/downloader/testdata/badterraformrc b/pkg/downloader/testdata/badterraformrc
new file mode 100644
index 000000000..822a7a18d
--- /dev/null
+++ b/pkg/downloader/testdata/badterraformrc
@@ -0,0 +1,2 @@
+credentials "doesntmatter" {
+  token = ""
\ No newline at end of file
diff --git a/pkg/downloader/testdata/terraformrc b/pkg/downloader/testdata/terraformrc
new file mode 100644
index 000000000..b557be412
--- /dev/null
+++ b/pkg/downloader/testdata/terraformrc
@@ -0,0 +1,3 @@
+credentials "app.terraform.io" {
+  token = "invalidusertoken"
+}
\ No newline at end of file