Repository: serverless/serverless (JavaScript)
Task type: Bug Fix

Issue:
Variable resolution broken in 1.30.0

# This is a Bug Report

## Description

For bug reports:
* What went wrong?

Variable are being replaced several times in CFN generation.

`Resource: "arn:aws:dynamodb:${opt:region, self:provider.region}:*:table/${self:provider.environment.SURFACES_TABLE}"`

 is becoming 

`"Resource": "arn:aws:dynamodb:eu-west-1:*:table/heat-exchanger-simulator-heat-exchanger-simulator-arn:aws:dynamodb:eu-west-1:*:table/heat-exchanger-simulator-heat-exchanger-simulator-dev-surfaces-surfaces-surfaces-surfaces"` 

in the generated CFN rather than 

`arn:aws:dynamodb:eu-west-1:*:table/heat-exchanger-simulator-dev-surfaces`

* What did you expect should have happened?
`arn:aws:dynamodb:eu-west-1:*:table/heat-exchanger-simulator-dev-surfaces`
* What was the config you used?
serverless 1.30.0 - 1.29.2 is not affected
* What stacktrace or error message from your provider did you see?

For feature proposals:
* What is the use case that should be solved. The more detail you describe this in the easier it is to understand for us.
* If there is additional config how would it look

Similar or dependent issues:
* #5156


## Additional Data

* ***Serverless Framework Version you're using***: 1.30.0
* ***Operating System***: ubuntu, debian, mac
* ***Stack Trace***: n/a
* ***Provider Error messages***: n/a

Variable interpolation fallback broken
# This is a Bug Report

## Description

With a nested variable interpolation with fallbacks, they seem to evaluate to false no matter what. I.e.:  `${self:custom.foo.${opt:stage, self:provider.stage}, 'shouldNotSee'}` will *always* result in `shouldNotSee`. This is a regression introduced in 1.30.

A replication is here: https://github.com/Asherlc/serverless-interpolation-bug

Run `yarn serverless print --stage production` to see the issue. Reverting to 1.29 solves the issue.

* ***Serverless Framework Version you're using***: 1.30
* ***Operating System***: macOS 10.13.6
* ***Stack Trace***: N/A
* ***Provider Error messages***: N/A


Discussion:
We're experiencing the same issue after upgrading to Serverless 1.30.
We avoided this problem replacing `${opt:region, self:provider.region}` with `${self:provider.region}` since we deploy only in one region.
We also experienced the same problem using `${opt:stage, self:provider.stage}`, replaced again with `${self:provider.stage}`.
There is probably a bug with that operator.
I think this is a duplicate of #5205 (or at least two of us following up on that issue have the issue you're seeing)
Looking into this.  Apologies for the issue.
Can confirm, I am seeing this exact behavior.  I just upgraded from 1.29 to 1.30 and all my variables based on this interpolation have ceased to function properly.  I'll have to revert to 1.29.
Was this defect introduced in #5156 ?
I think the same change(probably #5156) also introduced this issue I'm seeing:

`serverless.yml`:
```yaml
service:
  name: iam-error-test
  publish: false
custom:
  region: us-east-1
  stage: ${opt:stage, self:provider.stage}
provider:
  name: aws
  # Prevents serverless from trying to sub out ${AWS::blah}
  variableSyntax: "\\${((?!AWS)[ ~:a-zA-Z0-9._'\",\\-\\/\\(\\)]+?)}"
  region: ${self:custom.region}
  versionFunctions: false
  stage: ${env:USER, 'dev'}
  runtime: python3.6
  iamRoleStatements:
    - Effect: Allow
      Action:
        - lambda:InvokeFunction
      # This is effectively saying:
      # Resource: arn:aws:lambda:${self:custom.region}:${AWS::AccountId}:function:*
      Resource:
        - {'Fn::Join': [':', ['arn:aws:lambda', {Ref: 'AWS::Region'}, {Ref: 'AWS::AccountId'}, 'function:trading-${self:custom.${self:custom.stage}.stage, self:custom.stage}-*']]}

functions:
  test:
    handler: handlers.test
    events:
      - http:
          path: test
          method: get

```

`sls print` with 1.29.2 & older:
```yaml
service:
  name: iam-error-test
  publish: false
custom:
  region: us-east-1
  stage: dschep
provider:
  stage: dschep
  region: us-east-1
  name: aws
  versionFunctions: false
  runtime: python3.6
  iamRoleStatements:
    - Effect: Allow
      Action:
        - 'lambda:InvokeFunction'
      Resource:
        - 'Fn::Join':
            - ':'
            - - 'arn:aws:lambda'
              - Ref: 'AWS::Region'
              - Ref: 'AWS::AccountId'
              - 'function:trading-dschep-*'
  variableSyntax: '\${((?!AWS)[ ~:a-zA-Z0-9._''",\-\/\(\)]+?)}'
functions:
  test:
    handler: handlers.test
    events:
      - http:
          path: test
          method: get

```
`sls print` with 1.30.0:
```yaml
service:
  name: iam-error-test
  publish: false
custom:
  region: us-east-1
  stage: dschep
provider:
  stage: dschep
  region: us-east-1
  name: aws
  versionFunctions: false
  runtime: python3.6
  iamRoleStatements:
    - Effect: Allow
      Action:
        - 'lambda:InvokeFunction'
      Resource:
        - 'Fn::Join':
            - ':'
            - - 'arn:aws:lambda'
              - Ref: 'AWS::Region'
              - Ref: 'AWS::AccountId'
              - >-
                function:trading-function:trading-function:trading-function:trading-dschep-*-*-*-*
  variableSyntax: '\${((?!AWS)[ ~:a-zA-Z0-9._''",\-\/\(\)]+?)}'
functions:
  test:
    handler: handlers.test
    events:
      - http:
          path: test
          method: get

```


where the `-*-*-*-*` is obviously wrong and should just be `-*`

EDIT. oh and the weird `trading-function:trading-function:trading-function` crap too
Just updated to 1.30.0; deploy worked fine without --stage argument, but things got wonky after giving `--stage qa`.

Here is a snippet from my serverless.yaml:

```
service: service-name

provider:
  stage: ${opt:stage, 'dev'}
  environment:
    TABLE_NAME: ${self:service}-${opt:stage, self:provider.stage}-table

resources:
  Description: CloudFormation stack for ${self:service}-${opt:stage, self:provider.stage}.
```

The TABLE_NAME with the `--stage qa` argument became `service-name-service-name-qa-table-table`, also the Description had repeated content.

The deploy worked fine after reverting to 1.29.2 version.
That's odd @bugbreaka, it's the same duplication issue I'm seeing, but I'm not using `-s`.
Looking into this.  Apologies for the issue.
This occurs because although it does in most cases, in others `match.match != property` at https://github.com/serverless/serverless/blob/master/lib/classes/Variables.js#L304 . Coding a test and a fix.

To explain... The `property` value is used at https://github.com/serverless/serverless/blob/master/lib/classes/Variables.js#L457 to form a replacement in case of deep variable discovery.  However, in this case, prior to the error, the property is `${self:custom.foo.${opt:stage, self:provider.stage}, 'shouldNotSee'}` but `${opt:stage, self:provider.stage}` is being replaced.

Solution patch:
diff --git a/lib/classes/Variables.js b/lib/classes/Variables.js
index 8baa423b699..6411f043f70 100644
--- a/lib/classes/Variables.js
+++ b/lib/classes/Variables.js
@@ -301,7 +301,7 @@ class Variables {
    * @returns {Promise[]} Promises for the eventual populated values of the given matches
    */
   populateMatches(matches, property) {
-    return _.map(matches, (match) => this.splitAndGet(match.variable, property));
+    return _.map(matches, (match) => this.splitAndGet(match, property));
   }
   /**
    * Render the given matches and their associated results to the given value
@@ -354,14 +354,15 @@ class Variables {
   /**
    * Split the cleaned variable string containing one or more comma delimited variables and get a
    * final value for the entirety of the string
-   * @param varible The variable string to split and get a final value for
+   * @param match The regex match containing both the variable string to split and get a final
+   * value for as well as the originally matched string
    * @param property The original property string the given variable was extracted from
    * @returns {Promise} A promise resolving to the final value of the given variable
    */
-  splitAndGet(variable, property) {
-    const parts = this.splitByComma(variable);
+  splitAndGet(match, property) {
+    const parts = this.splitByComma(match.variable);
     if (parts.length > 1) {
-      return this.overwrite(parts, property);
+      return this.overwrite(parts, match.match, property);
     }
     return this.getValueFromSource(parts[0], property);
   }
@@ -440,11 +441,12 @@ class Variables {
   /**
    * Resolve the given variable string that expresses a series of fallback values in case the
    * initial values are not valid, resolving each variable and resolving to the first valid value.
-   * @param variableStringsString The overwrite string of variables to populate and choose from.
+   * @param variableStrings The overwrite variables to populate and choose from.
+   * @param variableMatch The original string from which the variables were extracted.
    * @returns {Promise.<TResult>|*} A promise resolving to the first validly populating variable
    *  in the given variable strings string.
    */
-  overwrite(variableStrings, propertyString) {
+  overwrite(variableStrings, variableMatch, propertyString) {
     const variableValues = variableStrings.map(variableString =>
       this.getValueFromSource(variableString, propertyString));
     const validValue = value => (
@@ -454,14 +456,15 @@ class Variables {
     );
     return BbPromise.all(variableValues)
       .then(values => {
-        let deepPropertyString = propertyString;
+        let deepPropertyString = variableMatch;
         let deepProperties = 0;
         values.forEach((value, index) => {
-          if (_.isString(value) && value.match(this.deepRefSyntax)) {
+          if (_.isString(value) && value.match(this.variableSyntax)) {
             deepProperties += 1;
+            const deepVariable = this.makeDeepVariable(value);
             deepPropertyString = deepPropertyString.replace(
               variableStrings[index],
-              this.cleanVariable(value));
+              this.cleanVariable(deepVariable));
           }
         });
         return deepProperties > 0 ?


Test patch:
diff --git a/lib/classes/Variables.test.js b/lib/classes/Variables.test.js
index 3d6cce0ba7d..19c68a8e324 100644
--- a/lib/classes/Variables.test.js
+++ b/lib/classes/Variables.test.js
@@ -600,6 +600,40 @@ describe('Variables', () => {
         return serverless.variables.populateObject(service.custom)
           .should.become(expected);
       });
+      it('should handle embedded deep variable replacements in overrides', () => {
+        service.custom = {
+          foo: 'bar',
+          val0: 'foo',
+          val1: '${self:custom.val0, "fallback 1"}',
+          val2: '${self:custom.${self:custom.val0, self:custom.val1}, "fallback 2"}',
+        };
+        const expected = {
+          foo: 'bar',
+          val0: 'foo',
+          val1: 'foo',
+          val2: 'bar',
+        };
+        return serverless.variables.populateObject(service.custom)
+          .should.become(expected);
+      });
+      it('should deal with overwites that reference embedded deep references', () => {
+        service.custom = {
+          val0: 'val',
+          val1: 'val0',
+          val2: '${self:custom.val1}',
+          val3: '${self:custom.${self:custom.val2}, "fallback"}',
+          val4: '${self:custom.val3, self:custom.val3}',
+        };
+        const expected = {
+          val0: 'val',
+          val1: 'val0',
+          val2: 'val0',
+          val3: 'val',
+          val4: 'val',
+        };
+        return serverless.variables.populateObject(service.custom)
+          .should.become(expected);
+      });
       it('should handle deep variables regardless of custom variableSyntax', () => {
         service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._\\\'",\\-\\/\\(\\)]+?)}}';
         serverless.variables.loadVariableSyntax();
@@ -990,6 +1024,11 @@ module.exports = {
   });
 
   describe('#overwrite()', () => {
+    beforeEach(() => {
+      serverless.service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}}';
+      serverless.variables.loadVariableSyntax();
+      delete serverless.service.provider.variableSyntax;
+    });
     it('should overwrite undefined and null values', () => {
       const getValueFromSourceStub = sinon.stub(serverless.variables, 'getValueFromSource');
       getValueFromSourceStub.onCall(0).resolves(undefined);