Repository: VictoriaMetrics/operator

Issue:
Allow `aws.*` configuration inside ` remoteWrite`
As a user of the operator
I would like to configure IAM based authentication inside my `remoteWrite` configuration
so that I don't need to (ab-)use `extraArgs` to use it.

The `vmagent` supports using IAM authentication by setting (for example) `-remoteWrite.aws.roleARN` et.al but the operator does not accept it as parameter yet. While it would be possible to use `extraArgs` to set these parameters, if used together with (for example) `inlineUrlRelabelConfig` one would need to generate a configmap outside of the general operator (helm) configuration and attach it to the agent. It would be better to have everything in one place.

Discussion:
@Haleygo: Hello @5nafu .
> It would be better to have everything in one place.

What do you mean by "have everything in one place"? Do you have any perferred proposal?
@5nafu: Hi @Haleygo,

Currently, if you have one or multiple remote-writes with at lease one IAM auth, your vmagent object might look like (only the relevant parts):

```yaml
remoteWrite:
  - url: http://vmsingle-victoria-stack.victoria.svc:8429/api/v1/write
  - url: https://vmingest.basic.auth.url/api/v1/write
    basicAuth:
      password:
        key: password
        name: victoria-credentials
      username:
        key: username
        name: victoria-credentials
    inlineUrlRelabelConfig:
    - action: labeldrop
      regex: source_.*|destination_service_.*|destination_can.*|destination_principal
    # ... 
  - url: https://victoria.i.am.auth.url/insert/123/prometheus/api/v1/write
    inlineUrlRelabelConfig:
    - action: drop
      regex: ^kube_.*;kubecost-cost-analyzer$
      source_labels:
      - __name__
      - job
    # ...
extraArgs:
    # ...
    remoteWrite.aws.region: ',,eu-central-1'
    remoteWrite.aws.roleARN: ',,arn:aws:iam::AccountID:role/Role'
    remoteWrite.aws.service: ',,execute-api'
    remoteWrite.aws.useSigv4: false,false,true
```

Notice the need to add the appropriate amount of comma (and boolean values) for each remote write. It is even worse when using the helm chart, as there will be an additional "local" write that the user does not configure.

It would be better for the user if one could use a configuration similar to the `basicAuth` like:

```yaml
remoteWrite:
  - url: http://vmsingle-victoria-stack.victoria.svc:8429/api/v1/write
  - url: https://vmingest.basic.auth.url/api/v1/write
    basicAuth:
      password:
        key: password
        name: victoria-credentials
      username:
        key: username
        name: victoria-credentials
    inlineUrlRelabelConfig:
    - action: labeldrop
      regex: source_.*|destination_service_.*|destination_can.*|destination_principal
    # ... 
  - url: https://victoria.i.am.auth.url/insert/123/prometheus/api/v1/write
    aws:
      region: 'eu-central-1'
      roleARN: 'arn:aws:iam::AccountID:role/Role'
      service: 'execute-api'
      useSigv4: true
    inlineUrlRelabelConfig:
    - action: drop
      regex: ^kube_.*;kubecost-cost-analyzer$
      source_labels:
      - __name__
      - job
    # ...
``` @f41gh7: I agree, your example with separate `aws` spec looks better.
 I think we should add more configuration params to the remoteWrite defintion. It makes configuration easy to understand without extraArgs comma-madness syntax. @f41gh7: The feature was included into [v0.61.0](https://docs.victoriametrics.com/operator/changelog/#v0610) release

 However, AWS credentials cannot be configured via YAML. It's only possible to use any kind of native AWS sdk authorization. 

Solution patch:
diff --git a/api/operator/v1beta1/vmagent_types.go b/api/operator/v1beta1/vmagent_types.go
index 6cf114c1..ba0f7c41 100644
--- a/api/operator/v1beta1/vmagent_types.go
+++ b/api/operator/v1beta1/vmagent_types.go
@@ -443,6 +443,22 @@ type VMAgentRemoteWriteSettings struct {
 	UseMultiTenantMode bool `json:"useMultiTenantMode,omitempty"`
 }
 
+// AWS defines AWS cloud auth specific params
+type AWS struct {
+	// EC2Endpoint is an optional AWS EC2 API endpoint to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set
+	EC2Endpoint string `json:"ec2Endpoint,omitempty"`
+	// Region is an optional AWS region to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set
+	Region string `json:"region,omitempty"`
+	// RoleARN is an optional AWS region to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set
+	RoleARN string `json:"roleARN,omitempty"`
+	// Service is an optional AWS Service to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set
+	Service string `json:"service,omitempty"`
+	// STSEndpoint is an optional AWS STS API endpoint to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set
+	STSEndpoint string `json:"stsEndpoint,omitempty"`
+	// UseSigv4 enables SigV4 request signing for the corresponding -remoteWrite.url
+	UseSigv4 bool `json:"useSigv4,omitempty"`
+}
+
 // VMAgentRemoteWriteSpec defines the remote storage configuration for VmAgent
 // +k8s:openapi-gen=true
 type VMAgentRemoteWriteSpec struct {
@@ -494,6 +510,8 @@ type VMAgentRemoteWriteSpec struct {
 	// ProxyURL for -remoteWrite.url. Supported proxies: http, https, socks5. Example: socks5://proxy:1234
 	// +optional
 	ProxyURL *string `json:"proxyURL,omitempty"`
+	// AWS describes params specific to AWS cloud
+	AWS *AWS `json:"aws,omitempty"`
 }
 
 // AsConfigMapKey key for kubernetes configmap
diff --git a/api/operator/v1beta1/zz_generated.deepcopy.go b/api/operator/v1beta1/zz_generated.deepcopy.go
index 48210785..6a144def 100644
--- a/api/operator/v1beta1/zz_generated.deepcopy.go
+++ b/api/operator/v1beta1/zz_generated.deepcopy.go
@@ -62,6 +62,21 @@ func (in *APIServerConfig) DeepCopy() *APIServerConfig {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *AWS) DeepCopyInto(out *AWS) {
+	*out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AWS.
+func (in *AWS) DeepCopy() *AWS {
+	if in == nil {
+		return nil
+	}
+	out := new(AWS)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *AdditionalServiceSpec) DeepCopyInto(out *AdditionalServiceSpec) {
 	*out = *in
@@ -3909,6 +3924,11 @@ func (in *VMAgentRemoteWriteSpec) DeepCopyInto(out *VMAgentRemoteWriteSpec) {
 		*out = new(string)
 		**out = **in
 	}
+	if in.AWS != nil {
+		in, out := &in.AWS, &out.AWS
+		*out = new(AWS)
+		**out = **in
+	}
 }
 
 // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new VMAgentRemoteWriteSpec.
diff --git a/config/crd/overlay/crd.yaml b/config/crd/overlay/crd.yaml
index e352eca1..d0448e49 100644
--- a/config/crd/overlay/crd.yaml
+++ b/config/crd/overlay/crd.yaml
@@ -6204,6 +6204,39 @@ spec:
                   description: VMAgentRemoteWriteSpec defines the remote storage configuration
                     for VmAgent
                   properties:
+                    aws:
+                      description: AWS describes params specific to AWS cloud
+                      properties:
+                        ec2Endpoint:
+                          description: EC2Endpoint is an optional AWS EC2 API endpoint
+                            to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4
+                            is set
+                          type: string
+                        region:
+                          description: Region is an optional AWS region to use for
+                            the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4
+                            is set
+                          type: string
+                        roleARN:
+                          description: RoleARN is an optional AWS region to use for
+                            the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4
+                            is set
+                          type: string
+                        service:
+                          description: Service is an optional AWS Service to use for
+                            the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4
+                            is set
+                          type: string
+                        stsEndpoint:
+                          description: STSEndpoint is an optional AWS STS API endpoint
+                            to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4
+                            is set
+                          type: string
+                        useSigv4:
+                          description: UseSigv4 enables SigV4 request signing for
+                            the corresponding -remoteWrite.url
+                          type: boolean
+                      type: object
                     basicAuth:
                       description: BasicAuth allow an endpoint to authenticate over
                         basic authentication
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index cf436105..4d129241 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -16,6 +16,7 @@ aliases:
 * Dependency: [vmoperator](https://docs.victoriametrics.com/operator/): Updated default versions for VL apps to [v1.25.0](https://github.com/VictoriaMetrics/VictoriaLogs/releases/tag/v1.25.0).
 
 * FEATURE: [vmanomaly](https://docs.victoriametrics.com/anomaly-detection/): update shards concurrently. See [#1434](https://github.com/VictoriaMetrics/operator/issues/1434).
+* FEATURE: [vmagent](https://docs.victoriametrics.com/operator/resources/vmagent/): added AWS section to VMAgent remoteWrite spec. See [#928](https://github.com/VictoriaMetrics/operator/issues/928).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/operator/resources/vmagent/): introduce global scrape config for VMAgent. See [#1179](https://github.com/VictoriaMetrics/operator/issues/1179).
 
 ## [v0.60.2](https://github.com/VictoriaMetrics/operator/releases/tag/v0.60.2)
diff --git a/docs/api.md b/docs/api.md
index 0e1469ad..b6616d1c 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -672,6 +672,24 @@ Appears in: [VMAgentSpec](#vmagentspec)
 | tlsConfig<a href="#apiserverconfig-tlsconfig" id="apiserverconfig-tlsconfig">#</a><br/>_[TLSConfig](#tlsconfig)_ | _(Optional)_<br/>TLSConfig Config to use for accessing apiserver. |
 
 
+#### AWS
+
+
+
+AWS defines AWS cloud auth specific params
+
+Appears in: [VMAgentRemoteWriteSpec](#vmagentremotewritespec)
+
+| Field | Description |
+| --- | --- |
+| ec2Endpoint<a href="#aws-ec2endpoint" id="aws-ec2endpoint">#</a><br/>_string_ | _(Required)_<br/>EC2Endpoint is an optional AWS EC2 API endpoint to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set |
+| region<a href="#aws-region" id="aws-region">#</a><br/>_string_ | _(Required)_<br/>Region is an optional AWS region to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set |
+| roleARN<a href="#aws-rolearn" id="aws-rolearn">#</a><br/>_string_ | _(Required)_<br/>RoleARN is an optional AWS region to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set |
+| service<a href="#aws-service" id="aws-service">#</a><br/>_string_ | _(Required)_<br/>Service is an optional AWS Service to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set |
+| stsEndpoint<a href="#aws-stsendpoint" id="aws-stsendpoint">#</a><br/>_string_ | _(Required)_<br/>STSEndpoint is an optional AWS STS API endpoint to use for the corresponding -remoteWrite.url if -remoteWrite.aws.useSigv4 is set |
+| useSigv4<a href="#aws-usesigv4" id="aws-usesigv4">#</a><br/>_boolean_ | _(Required)_<br/>UseSigv4 enables SigV4 request signing for the corresponding -remoteWrite.url |
+
+
 #### AdditionalServiceSpec
 
 
@@ -2805,6 +2823,7 @@ Appears in: [VMAgentSpec](#vmagentspec)
 
 | Field | Description |
 | --- | --- |
+| aws<a href="#vmagentremotewritespec-aws" id="vmagentremotewritespec-aws">#</a><br/>_[AWS](#aws)_ | _(Required)_<br/>AWS describes params specific to AWS cloud |
 | basicAuth<a href="#vmagentremotewritespec-basicauth" id="vmagentremotewritespec-basicauth">#</a><br/>_[BasicAuth](#basicauth)_ | _(Optional)_<br/>BasicAuth allow an endpoint to authenticate over basic authentication |
 | bearerTokenSecret<a href="#vmagentremotewritespec-bearertokensecret" id="vmagentremotewritespec-bearertokensecret">#</a><br/>_[SecretKeySelector](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#secretkeyselector-v1-core)_ | _(Optional)_<br/>Optional bearer auth token to use for -remoteWrite.url |
 | forceVMProto<a href="#vmagentremotewritespec-forcevmproto" id="vmagentremotewritespec-forcevmproto">#</a><br/>_boolean_ | _(Optional)_<br/>ForceVMProto forces using VictoriaMetrics protocol for sending data to -remoteWrite.url |
diff --git a/internal/controller/operator/factory/vmagent/vmagent.go b/internal/controller/operator/factory/vmagent/vmagent.go
index 8086c2d6..9a66f470 100644
--- a/internal/controller/operator/factory/vmagent/vmagent.go
+++ b/internal/controller/operator/factory/vmagent/vmagent.go
@@ -1123,6 +1123,12 @@ func buildRemoteWrites(cr *vmv1beta1.VMAgent, ac *build.AssetsCache) ([]string,
 	maxDiskUsagePerURL := remoteFlag{flagSetting: "-remoteWrite.maxDiskUsagePerURL="}
 	forceVMProto := remoteFlag{flagSetting: "-remoteWrite.forceVMProto="}
 	proxyURL := remoteFlag{flagSetting: "-remoteWrite.proxyURL="}
+	awsEC2Endpoint := remoteFlag{flagSetting: "-remoteWrite.aws.ec2Endpoint="}
+	awsRegion := remoteFlag{flagSetting: "-remoteWrite.aws.region="}
+	awsRoleARN := remoteFlag{flagSetting: "-remoteWrite.aws.roleARN="}
+	awsService := remoteFlag{flagSetting: "-remoteWrite.aws.service="}
+	awsSTSEndpoint := remoteFlag{flagSetting: "-remoteWrite.aws.stsEndpoint="}
+	awsUseSigv4 := remoteFlag{flagSetting: "-remoteWrite.aws.useSigv4="}
 
 	maxDiskUsage := defaultMaxDiskUsage
 	if cr.Spec.RemoteWriteSettings != nil && cr.Spec.RemoteWriteSettings.MaxDiskUsagePerURL != nil {
@@ -1268,6 +1274,41 @@ func buildRemoteWrites(cr *vmv1beta1.VMAgent, ac *build.AssetsCache) ([]string,
 		oauth2ClientID.flagSetting += fmt.Sprintf("%s,", oaclientID)
 		oauth2Scopes.flagSetting += fmt.Sprintf("%s,", oascopes)
 
+		var ec2Endpoint, region, service, roleARN, stsEndpoint string
+		var useSigv4 bool
+		if rws.AWS != nil {
+			if len(rws.AWS.EC2Endpoint) > 0 {
+				ec2Endpoint = rws.AWS.EC2Endpoint
+				awsEC2Endpoint.isNotNull = true
+			}
+			if len(rws.AWS.Region) > 0 {
+				region = rws.AWS.Region
+				awsRegion.isNotNull = true
+			}
+			if len(rws.AWS.RoleARN) > 0 {
+				roleARN = rws.AWS.RoleARN
+				awsRoleARN.isNotNull = true
+			}
+			if len(rws.AWS.Service) > 0 {
+				service = rws.AWS.Service
+				awsService.isNotNull = true
+			}
+			if len(rws.AWS.STSEndpoint) > 0 {
+				stsEndpoint = rws.AWS.STSEndpoint
+				awsSTSEndpoint.isNotNull = true
+			}
+			if rws.AWS.UseSigv4 {
+				useSigv4 = rws.AWS.UseSigv4
+				awsUseSigv4.isNotNull = true
+			}
+		}
+		awsEC2Endpoint.flagSetting += fmt.Sprintf("%s,", ec2Endpoint)
+		awsRegion.flagSetting += fmt.Sprintf("%s,", region)
+		awsRoleARN.flagSetting += fmt.Sprintf("%s,", roleARN)
+		awsService.flagSetting += fmt.Sprintf("%s,", service)
+		awsSTSEndpoint.flagSetting += fmt.Sprintf("%s,", stsEndpoint)
+		awsUseSigv4.flagSetting += fmt.Sprintf("%v,", useSigv4)
+
 		var dedupIntVal, streamConfVal string
 		var keepInputVal, dropInputVal, ignoreOldSamples, enableWindows bool
 		var ignoreFirstIntervalsVal int
@@ -1341,6 +1382,7 @@ func buildRemoteWrites(cr *vmv1beta1.VMAgent, ac *build.AssetsCache) ([]string,
 	remoteArgs = append(remoteArgs, headers, authPasswordFile)
 	remoteArgs = append(remoteArgs, streamAggrConfig, streamAggrKeepInput, streamAggrDedupInterval, streamAggrDropInput, streamAggrDropInputLabels, streamAggrIgnoreFirstIntervals, streamAggrIgnoreOldSamples, streamAggrEnableWindows)
 	remoteArgs = append(remoteArgs, maxDiskUsagePerURL, forceVMProto)
+	remoteArgs = append(remoteArgs, awsEC2Endpoint, awsRegion, awsRoleARN, awsService, awsSTSEndpoint, awsUseSigv4)
 
 	for _, remoteArgType := range remoteArgs {
 		if remoteArgType.isNotNull {



Test patch:
diff --git a/internal/controller/operator/factory/vmagent/vmagent_test.go b/internal/controller/operator/factory/vmagent/vmagent_test.go
index 6bc26608..86cdf59b 100644
--- a/internal/controller/operator/factory/vmagent/vmagent_test.go
+++ b/internal/controller/operator/factory/vmagent/vmagent_test.go
@@ -1281,6 +1281,48 @@ func TestBuildRemoteWrites(t *testing.T) {
 				`-remoteWrite.url=localhost:8428,localhost:8429,localhost:8430,localhost:8431,localhost:8432`,
 			},
 		},
+		{
+			name: "test with aws",
+			cr: &vmv1beta1.VMAgent{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "default-vmagent",
+					Namespace: "default",
+				},
+				Spec: vmv1beta1.VMAgentSpec{RemoteWrite: []vmv1beta1.VMAgentRemoteWriteSpec{
+					{
+						URL: "localhost:8429",
+						AWS: &vmv1beta1.AWS{
+							RoleARN: "arn:aws:iam::account:role/role-1",
+						},
+					},
+					{
+						URL: "localhost:8431",
+					},
+					{
+						URL: "localhost:8431",
+						AWS: &vmv1beta1.AWS{
+							UseSigv4: true,
+						},
+					},
+				}},
+			},
+			predefinedObjects: []runtime.Object{
+				&corev1.Secret{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "remote2-aws-secret",
+						Namespace: "default",
+					},
+					Data: map[string][]byte{
+						"accesskey": []byte("accesskeytoken"),
+					},
+				},
+			},
+			want: []string{
+				`-remoteWrite.aws.roleARN=arn:aws:iam::account:role/role-1,,`,
+				`-remoteWrite.aws.useSigv4=false,false,true`,
+				`-remoteWrite.url=localhost:8429,localhost:8431,localhost:8431`,
+			},
+		},
 		{
 			name: "test with proxyURL (one remote write with defaults)",
 			cr: &vmv1beta1.VMAgent{