Repository: serverless/serverless (JavaScript)
Date: 2019-11-08 19:14:50+00:00
Task type: Bug Fix

Issue:
AWS unable to use tags with custom partition name `aws-cn`
# Bug Report

AWS provider tagging fails on custom partition when a function is deployed as an API Gateway

## Description

<!-- Please use https://forum.serverless.com, StackOverflow or other forums for Q&A -->
<!-- Please answer ALL the question below. Otherwise we probably have to close the issue due to missing information -->

1. What did you do?
I am deploying an API Gateway with serverless in 2 different partitions: one "regular" AWS account (partition name being `aws`) and one specific to AWS China (partition name being `aws-cn`):
> serverless deploy --stage dev --region cn-northwest-1

2. What happened?
A Serverless Error occurs and breaks the deployment

3. What should've happened?
The deployment should finish without error

4. What's the content of your `serverless.yml` file?

```yaml
provider:
  name: aws
  stage: ${opt:stage, 'dev'}
  region: ${opt:region, 'us-east-1'} # overridden in CLI
  memorySize: 128
  timeout: 15
  # ...
  tags:
    foo: 'bar'

functions:
  hello-world:
    handler: functions/hello-world.handler
    runtime: nodejs10.x
    endpointType: regional # "regional" is required for AWS China, this can of course be set as a variable for different regions
    events:
      - http:
          path: /
          method: GET
          cors: true
```

5. What's the output you get when you use the `SLS_DEBUG=*` environment variable (e.g. `SLS_DEBUG=* serverless deploy`)
> Serverless: [AWS apigateway 400 4.774s 0 retries] tagResource({
>    resourceArn: 'arn:aws:apigateway:cn-northwest-1::/restapis/{XXXXXX}/stages/{YYYYYY}',
>    tags: { foo: 'bar' }
> })
> Serverless Error ---------------------------------------
 > ServerlessError: Expected ARN arn:aws-cn:apigateway:cn-northwest-1::/restapis/{XXXXXX}/stages/{YYYYYY} for Stage {YYYYYY} on RestApi {XXXXXX}, not arn:aws:apigateway:cn-northwest-1::/restapis/{XXXXXX}/stages/{YYYYYY}

The `tagResource()` in this case references the wrong partition name `arn:aws:[...]` instead of `arn:aws-cn:[...]`.

The deployment works when I comment out the `tags` config from the provider, which makes me think this is indeed a bug.

I just have a small doubt if I am missing some config for this to work, but I have not found anything related to that in the doc so far, so trying this bug report.

PS: I would be happy to submit a PR, if you wish to point me to the right direction to find the tagResource?
AWS unable to use tags with custom partition name `aws-cn`
# Bug Report

AWS provider tagging fails on custom partition when a function is deployed as an API Gateway

## Description

<!-- Please use https://forum.serverless.com, StackOverflow or other forums for Q&A -->
<!-- Please answer ALL the question below. Otherwise we probably have to close the issue due to missing information -->

1. What did you do?
I am deploying an API Gateway with serverless in 2 different partitions: one "regular" AWS account (partition name being `aws`) and one specific to AWS China (partition name being `aws-cn`):
> serverless deploy --stage dev --region cn-northwest-1

2. What happened?
A Serverless Error occurs and breaks the deployment

3. What should've happened?
The deployment should finish without error

4. What's the content of your `serverless.yml` file?

```yaml
provider:
  name: aws
  stage: ${opt:stage, 'dev'}
  region: ${opt:region, 'us-east-1'} # overridden in CLI
  memorySize: 128
  timeout: 15
  # ...
  tags:
    foo: 'bar'

functions:
  hello-world:
    handler: functions/hello-world.handler
    runtime: nodejs10.x
    endpointType: regional # "regional" is required for AWS China, this can of course be set as a variable for different regions
    events:
      - http:
          path: /
          method: GET
          cors: true
```

5. What's the output you get when you use the `SLS_DEBUG=*` environment variable (e.g. `SLS_DEBUG=* serverless deploy`)
> Serverless: [AWS apigateway 400 4.774s 0 retries] tagResource({
>    resourceArn: 'arn:aws:apigateway:cn-northwest-1::/restapis/{XXXXXX}/stages/{YYYYYY}',
>    tags: { foo: 'bar' }
> })
> Serverless Error ---------------------------------------
 > ServerlessError: Expected ARN arn:aws-cn:apigateway:cn-northwest-1::/restapis/{XXXXXX}/stages/{YYYYYY} for Stage {YYYYYY} on RestApi {XXXXXX}, not arn:aws:apigateway:cn-northwest-1::/restapis/{XXXXXX}/stages/{YYYYYY}

The `tagResource()` in this case references the wrong partition name `arn:aws:[...]` instead of `arn:aws-cn:[...]`.

The deployment works when I comment out the `tags` config from the provider, which makes me think this is indeed a bug.

I just have a small doubt if I am missing some config for this to work, but I have not found anything related to that in the doc so far, so trying this bug report.

PS: I would be happy to submit a PR, if you wish to point me to the right direction to find the tagResource?


Issue discussion:
Not only `aws-cn` but `aws-us-gov`.

Code can be found [here](https://github.com/serverless/serverless/blob/master/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.js#L287), and [here](https://github.com/serverless/serverless/blob/master/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.js#L235)

Solution could include pulling the [partition](https://github.com/serverless/serverless/blob/master/lib/plugins/aws/provider/awsProvider.js#L515) from [getAccountInfo()](https://github.com/serverless/serverless/blob/master/lib/plugins/aws/provider/awsProvider.js#L508) and using whatever partition the user is deploying to.

I'm happy to create a PR
You seem to know what you are talking about much more than I do @maafk.
Pulling the partition seems very like the way forward to me conceptually :)
I would be very happy of such a PR if you have the time for it
Not only `aws-cn` but `aws-us-gov`.

Code can be found [here](https://github.com/serverless/serverless/blob/master/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.js#L287), and [here](https://github.com/serverless/serverless/blob/master/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.js#L235)

Solution could include pulling the [partition](https://github.com/serverless/serverless/blob/master/lib/plugins/aws/provider/awsProvider.js#L515) from [getAccountInfo()](https://github.com/serverless/serverless/blob/master/lib/plugins/aws/provider/awsProvider.js#L508) and using whatever partition the user is deploying to.

I'm happy to create a PR
You seem to know what you are talking about much more than I do @maafk.
Pulling the partition seems very like the way forward to me conceptually :)
I would be very happy of such a PR if you have the time for it

Solution patch:
diff --git a/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.js b/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.js
index a4c5da7435f..34d24e83731 100644
--- a/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.js
+++ b/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.js
@@ -38,7 +38,7 @@ module.exports = {
     this.apiGatewayDeploymentId = null;
     this.apiGatewayRestApiId = null;
 
-    return resolveAccountId
+    return resolveAccountInfo
       .call(this)
       .then(resolveRestApiId.bind(this))
       .then(() => {
@@ -80,9 +80,12 @@ module.exports = {
   },
 };
 
-function resolveAccountId() {
+function resolveAccountInfo() {
   // eslint-disable-next-line no-return-assign
-  return this.provider.getAccountId().then(id => (this.accountId = id));
+  return this.provider.getAccountInfo().then(account => {
+    this.accountId = account.accountId;
+    this.partition = account.partition;
+  });
 }
 
 function resolveApiGatewayResource(resources) {
@@ -203,6 +206,7 @@ function handleLogs() {
     const service = this.state.service.service;
     const stage = this.options.stage;
     const region = this.options.region;
+    const partition = this.partition;
     const logGroupName = `/aws/api-gateway/${service}-${stage}`;
 
     operations = [];
@@ -231,12 +235,7 @@ function handleLogs() {
     const accessLogging = logs.accessLogging == null ? true : logs.accessLogging;
 
     if (accessLogging) {
-      let resourceArn;
-      if (region.includes('gov')) {
-        resourceArn = `arn:aws-us-gov:logs:${region}:${this.accountId}:log-group:${logGroupName}`;
-      } else {
-        resourceArn = `arn:aws:logs:${region}:${this.accountId}:log-group:${logGroupName}`;
-      }
+      const resourceArn = `arn:${partition}:logs:${region}:${this.accountId}:log-group:${logGroupName}`;
       const destinationArn = {
         op: 'replace',
         path: '/accessLogSettings/destinationArn',
@@ -283,12 +282,8 @@ function handleTags() {
   const restApiId = this.apiGatewayRestApiId;
   const stageName = this.options.stage;
   const region = this.options.region;
-  let resourceArn;
-  if (region.includes('gov')) {
-    resourceArn = `arn:aws-us-gov:apigateway:${region}::/restapis/${restApiId}/stages/${stageName}`;
-  } else {
-    resourceArn = `arn:aws:apigateway:${region}::/restapis/${restApiId}/stages/${stageName}`;
-  }
+  const partition = this.partition;
+  const resourceArn = `arn:${partition}:apigateway:${region}::/restapis/${restApiId}/stages/${stageName}`;
 
   if (tagKeysToBeRemoved.length > 0) {
     this.apiGatewayUntagResourceParams.push({


Test patch:
diff --git a/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.test.js b/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.test.js
index 7c0ff165298..db646154e22 100644
--- a/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.test.js
+++ b/lib/plugins/aws/package/compile/events/apiGateway/lib/hack/updateStage.test.js
@@ -23,7 +23,7 @@ describe('#updateStage()', () => {
   let serverless;
   let options;
   let awsProvider;
-  let providerGetAccountIdStub;
+  let providerGetAccountInfoStub;
   let providerRequestStub;
   let context;
 
@@ -33,7 +33,10 @@ describe('#updateStage()', () => {
     options = { stage: 'dev', region: 'us-east-1' };
     awsProvider = new AwsProvider(serverless, options);
     serverless.setProvider('aws', awsProvider);
-    providerGetAccountIdStub = sinon.stub(awsProvider, 'getAccountId').resolves(123456);
+    providerGetAccountInfoStub = sinon.stub(awsProvider, 'getAccountInfo').resolves({
+      accountId: '123456',
+      partition: 'aws',
+    });
     providerRequestStub = sinon.stub(awsProvider, 'request');
     serverless.service.provider.compiledCloudFormationTemplate = {
       Resources: {
@@ -98,7 +101,7 @@ describe('#updateStage()', () => {
   });
 
   afterEach(() => {
-    awsProvider.getAccountId.restore();
+    awsProvider.getAccountInfo.restore();
     awsProvider.request.restore();
   });
 
@@ -136,7 +139,7 @@ describe('#updateStage()', () => {
         { op: 'replace', path: '/*/*/logging/loglevel', value: 'INFO' },
       ];
 
-      expect(providerGetAccountIdStub).to.be.calledOnce;
+      expect(providerGetAccountInfoStub).to.be.calledOnce;
       expect(providerRequestStub.args).to.have.length(5);
       expect(providerRequestStub.args[0][0]).to.equal('APIGateway');
       expect(providerRequestStub.args[0][1]).to.equal('getRestApis');
@@ -178,6 +181,11 @@ describe('#updateStage()', () => {
 
   it('should support gov regions', () => {
     options.region = 'us-gov-east-1';
+    awsProvider.getAccountInfo.restore();
+    providerGetAccountInfoStub = sinon.stub(awsProvider, 'getAccountInfo').resolves({
+      accountId: '123456',
+      partition: 'aws-us-gov',
+    });
     context.state.service.provider.tags = {
       'Containing Space': 'bar',
       'bar': 'high-priority',
@@ -212,7 +220,7 @@ describe('#updateStage()', () => {
         { op: 'replace', path: '/*/*/logging/loglevel', value: 'INFO' },
       ];
 
-      expect(providerGetAccountIdStub).to.be.calledOnce;
+      expect(providerGetAccountInfoStub).to.be.calledOnce;
       expect(providerRequestStub.args).to.have.length(5);
       expect(providerRequestStub.args[0][0]).to.equal('APIGateway');
       expect(providerRequestStub.args[0][1]).to.equal('getRestApis');
@@ -252,6 +260,86 @@ describe('#updateStage()', () => {
     });
   });
 
+  it('should support all partitions', () => {
+    options.region = 'cn-northwest-1';
+    awsProvider.getAccountInfo.restore();
+    providerGetAccountInfoStub = sinon.stub(awsProvider, 'getAccountInfo').resolves({
+      accountId: '123456',
+      partition: 'aws-cn',
+    });
+    context.state.service.provider.tags = {
+      'Containing Space': 'bar',
+      'bar': 'high-priority',
+    };
+    context.state.service.provider.stackTags = {
+      bar: 'low-priority',
+      num: 123,
+    };
+    context.state.service.provider.tracing = {
+      apiGateway: true,
+    };
+    context.state.service.provider.logs = {
+      restApi: true,
+    };
+
+    return updateStage.call(context).then(() => {
+      const patchOperations = [
+        { op: 'replace', path: '/tracingEnabled', value: 'true' },
+        {
+          op: 'replace',
+          path: '/accessLogSettings/destinationArn',
+          value: 'arn:aws-cn:logs:cn-northwest-1:123456:log-group:/aws/api-gateway/my-service-dev',
+        },
+        {
+          op: 'replace',
+          path: '/accessLogSettings/format',
+          value:
+            'requestId: $context.requestId, ip: $context.identity.sourceIp, caller: $context.identity.caller, user: $context.identity.user, requestTime: $context.requestTime, httpMethod: $context.httpMethod, resourcePath: $context.resourcePath, status: $context.status, protocol: $context.protocol, responseLength: $context.responseLength',
+        },
+        { op: 'replace', path: '/*/*/logging/dataTrace', value: 'true' },
+        { op: 'replace', path: '/*/*/logging/loglevel', value: 'INFO' },
+      ];
+
+      expect(providerGetAccountInfoStub).to.be.calledOnce;
+      expect(providerRequestStub.args).to.have.length(5);
+      expect(providerRequestStub.args[0][0]).to.equal('APIGateway');
+      expect(providerRequestStub.args[0][1]).to.equal('getRestApis');
+      expect(providerRequestStub.args[0][2]).to.deep.equal({
+        limit: 500,
+        position: undefined,
+      });
+      expect(providerRequestStub.args[1][0]).to.equal('APIGateway');
+      expect(providerRequestStub.args[1][1]).to.equal('getStage');
+      expect(providerRequestStub.args[1][2]).to.deep.equal({
+        restApiId: 'devRestApiId',
+        stageName: 'dev',
+      });
+      expect(providerRequestStub.args[2][0]).to.equal('APIGateway');
+      expect(providerRequestStub.args[2][1]).to.equal('updateStage');
+      expect(providerRequestStub.args[2][2]).to.deep.equal({
+        restApiId: 'devRestApiId',
+        stageName: 'dev',
+        patchOperations,
+      });
+      expect(providerRequestStub.args[3][0]).to.equal('APIGateway');
+      expect(providerRequestStub.args[3][1]).to.equal('tagResource');
+      expect(providerRequestStub.args[3][2]).to.deep.equal({
+        resourceArn: 'arn:aws-cn:apigateway:cn-northwest-1::/restapis/devRestApiId/stages/dev',
+        tags: {
+          'Containing Space': 'bar',
+          'bar': 'high-priority',
+          'num': '123',
+        },
+      });
+      expect(providerRequestStub.args[4][0]).to.equal('APIGateway');
+      expect(providerRequestStub.args[4][1]).to.equal('untagResource');
+      expect(providerRequestStub.args[4][2]).to.deep.equal({
+        resourceArn: 'arn:aws-cn:apigateway:cn-northwest-1::/restapis/devRestApiId/stages/dev',
+        tagKeys: ['old'],
+      });
+    });
+  });
+
   it('should perform default actions if settings are not configure', () => {
     context.state.service.provider.tags = {
       old: 'tag',
@@ -263,7 +351,7 @@ describe('#updateStage()', () => {
         { op: 'replace', path: '/*/*/logging/loglevel', value: 'OFF' },
       ];
 
-      expect(providerGetAccountIdStub).to.be.calledOnce;
+      expect(providerGetAccountInfoStub).to.be.calledOnce;
       expect(providerRequestStub.args).to.have.length(4);
       expect(providerRequestStub.args[0][0]).to.equal('APIGateway');
       expect(providerRequestStub.args[0][1]).to.equal('getRestApis');
@@ -324,7 +412,7 @@ describe('#updateStage()', () => {
         { op: 'replace', path: '/*/*/logging/loglevel', value: 'OFF' },
       ];
 
-      expect(providerGetAccountIdStub).to.be.calledOnce;
+      expect(providerGetAccountInfoStub).to.be.calledOnce;
       expect(providerRequestStub.args).to.have.length(6);
       expect(providerRequestStub.args[0][0]).to.equal('APIGateway');
       expect(providerRequestStub.args[0][1]).to.equal('getRestApis');
@@ -489,7 +577,7 @@ describe('#updateStage()', () => {
         { op: 'replace', path: '/*/*/logging/loglevel', value: 'INFO' },
       ];
 
-      expect(providerGetAccountIdStub).to.be.calledOnce;
+      expect(providerGetAccountInfoStub).to.be.calledOnce;
       expect(providerRequestStub.args).to.have.length(4);
       expect(providerRequestStub.args[0][0]).to.equal('APIGateway');
       expect(providerRequestStub.args[0][1]).to.equal('getRestApis');