Repository: serverless/serverless (JavaScript)
Date: 2018-05-02 18:57:39+00:00
Task type: Bug Fix

Issue:
Variables resolution broken with custom variableSyntax (${deep:0})
<!--
1. If you have a question and not a bug/feature request please ask it at http://forum.serverless.com
2. Please check if an issue already exists so there are no duplicates
3. Check out and follow our Guidelines: https://github.com/serverless/serverless/blob/master/CONTRIBUTING.md
4. Fill out the whole template so we have a good overview on the issue
5. Do not remove any section of the template. If something is not applicable leave it empty but leave it in the Issue
6. Please follow the template, otherwise we'll have to ask you to update it
-->

# This is a Bug Report

PR to follow.

## Description

For bug reports:
* What went wrong?
Users of a custom variable syntax are seeing variable population failures that leave behind `${deep:0...}` artifacts

* What did you expect should have happened?
The variables should render as they previous did

* What was the config you used?
Using this config:
```
service: new-service

provider:
  name: aws
  runtime: nodejs6.10
  variableSyntax: "\\${{([ ~:a-zA-Z0-9._\\'\",\\-\\/\\(\\)]+?)}}"

custom:
  my1stStage: ${{opt:stage}}
  my2ndStage: ${{self:custom.my1stStage}}
```

And running:
```
sls package -s DEV
```

One can reference `.serverless/serverless-state.json` and observe the following:
```
[...]
    "custom": {
      "my1stStage": "DEV",
      "my2ndStage": "${deep:0}"
    },
[...]
```

Very clearly, the expected value would be:
```
[...]
    "custom": {
      "my1stStage": "DEV",
      "my2ndStage": "DEV"
    },
[...]
```

The problem results from the `${deep:0}` variable not complying with the custom syntax.

* What stacktrace or error message from your provider did you see?

N/A

## Additional Data

* ***Serverless Framework Version you're using***: 1.27.0
* ***Operating System***: OSx
* ***Stack Trace***: N/A
* ***Provider Error messages***: N/A
Variables resolution broken with custom variableSyntax (${deep:0})
<!--
1. If you have a question and not a bug/feature request please ask it at http://forum.serverless.com
2. Please check if an issue already exists so there are no duplicates
3. Check out and follow our Guidelines: https://github.com/serverless/serverless/blob/master/CONTRIBUTING.md
4. Fill out the whole template so we have a good overview on the issue
5. Do not remove any section of the template. If something is not applicable leave it empty but leave it in the Issue
6. Please follow the template, otherwise we'll have to ask you to update it
-->

# This is a Bug Report

PR to follow.

## Description

For bug reports:
* What went wrong?
Users of a custom variable syntax are seeing variable population failures that leave behind `${deep:0...}` artifacts

* What did you expect should have happened?
The variables should render as they previous did

* What was the config you used?
Using this config:
```
service: new-service

provider:
  name: aws
  runtime: nodejs6.10
  variableSyntax: "\\${{([ ~:a-zA-Z0-9._\\'\",\\-\\/\\(\\)]+?)}}"

custom:
  my1stStage: ${{opt:stage}}
  my2ndStage: ${{self:custom.my1stStage}}
```

And running:
```
sls package -s DEV
```

One can reference `.serverless/serverless-state.json` and observe the following:
```
[...]
    "custom": {
      "my1stStage": "DEV",
      "my2ndStage": "${deep:0}"
    },
[...]
```

Very clearly, the expected value would be:
```
[...]
    "custom": {
      "my1stStage": "DEV",
      "my2ndStage": "DEV"
    },
[...]
```

The problem results from the `${deep:0}` variable not complying with the custom syntax.

* What stacktrace or error message from your provider did you see?

N/A

## Additional Data

* ***Serverless Framework Version you're using***: 1.27.0
* ***Operating System***: OSx
* ***Stack Trace***: N/A
* ***Provider Error messages***: N/A


Issue discussion:
Attached milestone 1.27.1 to get it out as hotfix as soon as it's released. Thanks @erikerikson for getting onto the problem 👍 
Attached milestone 1.27.1 to get it out as hotfix as soon as it's released. Thanks @erikerikson for getting onto the problem 👍 

Solution patch:
diff --git a/lib/classes/Variables.js b/lib/classes/Variables.js
index 77b48e4a5d2..28f62386ae7 100644
--- a/lib/classes/Variables.js
+++ b/lib/classes/Variables.js
@@ -729,7 +729,15 @@ class Variables {
     if (index < 0) {
       index = this.deep.push(variable) - 1;
     }
-    return `\${deep:${index}}`;
+    let variableContainer = variable;
+    let variableString = this.cleanVariable(variableContainer);
+    while (variableString.match(this.variableSyntax)) {
+      variableContainer = variableString;
+      variableString = this.cleanVariable(variableContainer);
+    }
+    return variableContainer
+      .replace(/\s/g, '')
+      .replace(variableString, `deep:${index}`);
   }
   appendDeepVariable(variable, subProperty) {
     return `${variable.slice(0, variable.length - 1)}.${subProperty}}`;


Test patch:
diff --git a/lib/classes/Variables.test.js b/lib/classes/Variables.test.js
index 55ae8f45e0a..c03bceeef19 100644
--- a/lib/classes/Variables.test.js
+++ b/lib/classes/Variables.test.js
@@ -586,6 +586,44 @@ describe('Variables', () => {
         return serverless.variables.populateObject(service.custom)
           .should.become(expected);
       });
+      it('should handle deep variables regardless of custom variableSyntax', () => {
+        service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._\\\'",\\-\\/\\(\\)]+?)}}';
+        serverless.variables.loadVariableSyntax();
+        delete service.provider.variableSyntax;
+        service.custom = {
+          my0thStage: 'DEV',
+          my1stStage: '${{self:custom.my0thStage}}',
+          my2ndStage: '${{self:custom.my1stStage}}',
+        };
+        const expected = {
+          my0thStage: 'DEV',
+          my1stStage: 'DEV',
+          my2ndStage: 'DEV',
+        };
+        return serverless.variables.populateObject(service.custom)
+          .should.become(expected);
+      });
+      it('should handle deep variables regardless of recursion into custom variableSyntax', () => {
+        service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._\\\'",\\-\\/\\(\\)]+?)}}';
+        serverless.variables.loadVariableSyntax();
+        delete service.provider.variableSyntax;
+        service.custom = {
+          my0thIndex: '0th',
+          my1stIndex: '1st',
+          my0thStage: 'DEV',
+          my1stStage: '${{self:custom.my${{self:custom.my0thIndex}}Stage}}',
+          my2ndStage: '${{self:custom.my${{self:custom.my1stIndex}}Stage}}',
+        };
+        const expected = {
+          my0thIndex: '0th',
+          my1stIndex: '1st',
+          my0thStage: 'DEV',
+          my1stStage: 'DEV',
+          my2ndStage: 'DEV',
+        };
+        return serverless.variables.populateObject(service.custom)
+          .should.become(expected);
+      });
       describe('file reading cases', () => {
         let tmpDirPath;
         beforeEach(() => {