AWS Opensearchとterraform-opensearch-providerのマスターユーザーの設定にハマった

背景
---
OpenSeachをAWS上で構成する際、インデックス管理やロールマッピングなどの設定を宣言的に管理できないかと思い、公式が出しているTerraform OpenSearch Providerを採用しました。

※ resourceの例
```tf
resource "opensearch_index" "test" {
  name               = "terraform-test"
  number_of_shards   = 1
  number_of_replicas = 1
  mappings           = <<EOF
{
  "people": {
    "_all": {
      "enabled": false
    },
    "properties": {
      "email": {
        "type": "text"
      }
    }
  }
}
EOF
}
```

OpenSearchの認証
---

AWS OpenSearchの認証フローは複雑になりやすく、Black Beltに乗っている認証ベストプラクティスは以下の図のようになります。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/1793316/ddd4417a-8b0b-ef3c-adb7-136545beb586.png)

https://pages.awscloud.com/rs/112-TZM-766/images/AWS-Black-Belt_2023_Amazon-OpenSearch-Service-Best-Practice-Security_0430_v1.pdf


今回はVPCに入れない構成だったため、認証+ドメインポリシー+きめ細やかなアクセス制御(FGAC)という3層フローです。

OpenSearchはFGACを有効にすると内部ユーザーを作成する必要があるのですが、ID+PWの方式はパスワードの管理が面倒だと感じIAM方式で認証設定しました。今回の本筋ではないので外しますがdashboardへのアクセスはcognitoを利用しています。

```tf
resource "aws_opensearch_domain" "opensearch_domain" {
  domain_name    = local.opensearch_domain_name
  engine_version = "OpenSearch_2.13"

  # <略>

  advanced_security_options {
    enabled                        = true
    internal_user_database_enabled = false

    master_user_options {
      master_user_arn = aws_iam_role.opensearch_master_user_role.arn
    }
  }
}
```

困ったこと
---

内部のadmin権限をIAM roleで指定しているため、terraform opensearch providerでもそのようにして権限を渡して上げる必要があります。

https://registry.terraform.io/providers/opensearch-project/opensearch/latest/docs#authentication

公式の例では以下のようにしてproviderを設定しています。
```tf
provider "opensearch" {
  url      = "https://search-foo-bar-pqrhr4w3u4dzervg41frow4mmy.us-east-1.es.amazonaws.com"
  username = "ausername"
  password = "apassword"
  # Must be disabled for basic auth
  sign_aws_requests = false
}
```

```tf
provider "opensearch" {
  url                         = "https://search-foo-bar-pqrhr4w3u4dzervg41frow4mmy.us-east-1.es.amazonaws.com"
  aws_assume_role_arn         = "arn:aws:iam::012345678901:role/rolename"
  aws_assume_role_external_id = "SecretID"
}
```

providerブロックに渡すパラメータは以下のようになってます。


| Parameter                  | Type    | Description                                                                                                                     |
|----------------------------|---------|---------------------------------------------------------------------------------------------------------------------------------|
| aws_access_key             | String  | The access key for use with AWS OpenSearch Service domains                                                                      |
| aws_assume_role_arn        | String  | Amazon Resource Name of an IAM Role to assume prior to making AWS API calls.                                                    |
| aws_assume_role_external_id| String  | External ID configured in the IAM policy of the IAM Role to assume prior to making AWS API calls.                               |
| aws_profile                | String  | The AWS profile for use with AWS OpenSearch Service domains                                                                     |
| aws_region                 | String  | The AWS region for use in signing of AWS OpenSearch requests. Must be specified to use AWS URL signing with a custom DNS domain.|
| aws_secret_key             | String  | The secret key for use with AWS OpenSearch Service domains                                                                      |
| aws_signature_service      | String  | AWS service name used in the credential scope of signed requests to OpenSearch.                                                 |
| aws_token                  | String  | The session token for use with AWS OpenSearch Service domains                                                                   |
| cacert_file                | String  | A Custom CA certificate                                                                                                         |
| client_cert_path           | String  | A X509 certificate to connect to OpenSearch                                                                                     |
| client_key_path            | String  | A X509 key to connect to OpenSearch                                                                                             |
| healthcheck                | Boolean | Set the client healthcheck option for the OpenSearch client. Designed for direct access to the cluster.                         |
| host_override              | String  | Sets the 'Host' header of requests and the 'ServerName' for certificate validation to this value.                               |
| insecure                   | Boolean | Disable SSL verification of API calls                                                                                           |
| opensearch_version         | String  | OpenSearch Version                                                                                                              |
| password                   | String  | Password to use to connect to OpenSearch using basic auth                                                                       |
| proxy                      | String  | Proxy URL to use for requests to OpenSearch.                                                                                    |
| sign_aws_requests          | Boolean | Enable signing of AWS OpenSearch requests. Must refer to AWS ES domain or aws_region must be specified explicitly.              |
| sniff                      | Boolean | Set the node sniffing option for the OpenSearch client. Won't work if nodes are not routable.                                   |
| token                      | String  | A bearer token or ApiKey for an Authorization header, e.g., Active Directory API key.                                           |
| token_name                 | String  | The type of token, usually ApiKey or Bearer                                                                                     |
| username                   | String  | Username to use to connect to OpenSearch using basic auth                                                                       |
| version_ping_timeout       | Number  | Version ping timeout in seconds 


これをみて以下のようにproviderブロックを書いたところ、
```tf
provider "opensearch" {
  url = module.opensearch.endpoint
  aws_region  = var.region
  sign_aws_requests  = true
  aws_access_key_id = var.aws_access_key_id
  aws_secret_key = var.aws_secret_key
  aws_assume_role_arn = module.opensearch.master_role_arn
}
```

- `Error: NoCredentialProviders: no valid providers`
- `Error: HTTP 403 Forbidden: Permission denied. Please ensure that the correct credentials are being used to access the cluster `
 
といったエラーが出ます。

原因
---

なぜかと思い色々調べたところ、Providerの実装をみてようやく答えがわかりました。

```go
func awsSession(region string, conf *ProviderConf, endpoint string) *awssession.Session {
	sessOpts := awsSessionOptions(region, endpoint)

	// 1. access keys take priority
	// 2. next is an assume role configuration
	// 3. followed by a profile (for assume role)
	// 4. let the default credentials provider figure out the rest (env, ec2, etc..)
	//
	// note: if #1 is chosen, then no further providers will be tested, since we've overridden the credentials with just a static provider
	if conf.awsAccessKeyId != "" {
		sessOpts.Config.Credentials = awscredentials.NewStaticCredentials(conf.awsAccessKeyId, conf.awsSecretAccessKey, conf.awsSessionToken)
	} else if conf.awsAssumeRoleArn != "" {
		if conf.awsAssumeRoleExternalID == "" {
			conf.awsAssumeRoleExternalID = ""
		}
		sessOpts.Config.Credentials = assumeRoleCredentials(region, conf.awsAssumeRoleArn, conf.awsAssumeRoleExternalID, conf.awsProfile, endpoint)
	} else if conf.awsProfile != "" {
		sessOpts.Profile = conf.awsProfile
	}
    // 略
```
最初のif~else ifをみて分かる通り、**access_key_idを明示的に与えたときはassumeroleしてくれないのです。**

特定のcredentialを用いてassume roleする場合、access_key_idは明示的に与えずに、profileを指定するか、環境変数に入れておく必要があるようです

```go
assumeRoleCredentials(region, conf.awsAssumeRoleArn, conf.awsAssumeRoleExternalID, conf.awsProfile, endpoint)
```


解決
---

今回は実行環境にTerraform Cloudを用いてたためvariableでaccess keyを渡していました。terraform cloudで~/.aws/credentialを変更するのは現実的ではないと思い環境変数として設定することで解決しました。