Repository: serverless/serverless (JavaScript)
Date: 2019-02-06 01:15:49+00:00
Task type: Bug Fix

Issue:
Custom variableSyntax with file variable resolution fails (${{deep:0})
# This is a Bug Report
## Description
* What went wrong?
Custom variable syntax and file variable resolutions results in faulty variables containing ${{deep:0...}

* What did you expect should have happened?
I expected the template using a custom variable syntax to work with ${file} variable resolution. And produce the following results.
```
$ sls print
service: template
provider:
  name: aws
  runtime: nodejs8.10
custom:
  settings:
    description: this is dummy
functions:
  hello:
    handler: handler.hello
    description: this is dummy
```
* What was the config you used?
```
# env.yml
description: this is dummy
```
 ```
# serverless.yml - Errorneous template with a custom variable syntax 
service: template

provider:
  name: aws
  runtime: nodejs8.10
  variableSyntax: "\\${{([ ~:a-zA-Z0-9._@\\'\",\\-\\/\\(\\)]+?)}}"

custom:
  settings: ${{file(./env.yml)}}

functions:
  hello:
    handler: handler.hello
    description: ${{self:custom.settings.description}}
```
```
# serverless.yml - Successful template without variable syntax 
service: template

provider:
  name: aws
  runtime: nodejs8.10

custom:
  settings: ${file(./env.yml)}

functions:
  hello:
    handler: handler.hello
    description: ${self:custom.settings.description}
```
```
# serverless.yml - Successful template with a custom variable syntax and selecting one variable with file 
service: template

provider:
  name: aws
  runtime: nodejs8.10
  variableSyntax: "\\${{([ ~:a-zA-Z0-9._@\\'\",\\-\\/\\(\\)]+?)}}"

custom:
  settings: ${{file(./env.yml):description}}

functions:
  hello:
    handler: handler.hello
    description: ${{self:custom.settings}}
```

* What stacktrace or error message from your provider did you see?
```
$ sls print
service: template
provider:
  name: aws
  runtime: nodejs8.10
  variableSyntax: '\${{([ ~:a-zA-Z0-9._@\''",\-\/\(\)]+?)}}'
custom:
  settings:
    description: this is dummy
functions:
  hello:
    handler: handler.hello
    description: '${{deep:0}.description}'
```
Similar or dependent issues:
* #4946
* #4953
* #4989

## Additional Data
@TassSinclair You mentioned this got resolved in #4989, is it still working for you? 
@erikerikson worked on these issues last year. Including him in the bug report in case he is interested in following this.

* ***Serverless Framework Version you're using***: 1.36.3
* ***Operating System***: Mac OS 10.14.2 

## Update 20190201
* Added example with a successful template using file and variable syntax but only selecting 1 variable.
* I should also mention that this issue is not a blocker for me as I managed to work around the issue with the successful template above. So as far as I am concerned you can prioritize this however you see fit. 



Issue discussion:
Thanks for the CC, sorry you're experiencing this issue!
Happening for me as well.

```
custom:
  bucket: "content-storage-${self:provider.stage}"
```

The issue seems to be with the stage variable I am using.

Actually, I just removed the variable syntax key and my problem vanished. This key was part of the template I used so I don't really know what it does.

```
provider:
  variableSyntax: '\${((env|self|opt|file|cf|s3)[:\(][ :a-zA-Z0-9._,\-\/\(\)]*?)}'
```
@logicminds, my guess would be that they're trying to make it possible to have `${AWS::foobar}` without tripping up serverless's variable system.
This is going sideways because of variable syntax assumptions at https://github.com/serverless/serverless/blob/3b9957f0725ac7ebf48d60e1c14d0cba096aa57c/lib/classes/Variables.js#L794 - confirmed fix incoming after crafting a test to avoid re-occurrence.

Solution patch:
diff --git a/lib/classes/Variables.js b/lib/classes/Variables.js
index aa52ac7a246..6d81ab4bfdc 100644
--- a/lib/classes/Variables.js
+++ b/lib/classes/Variables.js
@@ -791,7 +791,8 @@ class Variables {
       .replace(variableString, `deep:${index}`);
   }
   appendDeepVariable(variable, subProperty) {
-    return `${variable.slice(0, variable.length - 1)}.${subProperty}}`;
+    const variableString = this.cleanVariable(variable);
+    return variable.replace(variableString, `${variableString}.${subProperty}`);
   }
 
   /**


Test patch:
diff --git a/lib/classes/Variables.test.js b/lib/classes/Variables.test.js
index 97f8a527b13..e6e3dfdb1d4 100644
--- a/lib/classes/Variables.test.js
+++ b/lib/classes/Variables.test.js
@@ -835,6 +835,23 @@ describe('Variables', () => {
         return serverless.variables.populateObject(service.custom)
           .should.become(expected);
       });
+      it('should handle deep variable continuations regardless of custom variableSyntax', () => {
+        service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._@\\\'",\\-\\/\\(\\)]+?)}}';
+        serverless.variables.loadVariableSyntax();
+        delete service.provider.variableSyntax;
+        service.custom = {
+          my0thStage: { we: 'DEV' },
+          my1stStage: '${{self:custom.my0thStage}}',
+          my2ndStage: '${{self:custom.my1stStage.we}}',
+        };
+        const expected = {
+          my0thStage: { we: 'DEV' },
+          my1stStage: { we: 'DEV' },
+          my2ndStage: 'DEV',
+        };
+        return serverless.variables.populateObject(service.custom)
+          .should.become(expected);
+      });
       it('should handle deep variables regardless of recursion into custom variableSyntax', () => {
         service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._@\\\'",\\-\\/\\(\\)]+?)}}';
         serverless.variables.loadVariableSyntax();