Repository: serverless/serverless (JavaScript)
Date: 2018-07-24 21:55:47+00:00
Task type: Bug Fix

Issue:
Nested variable expansion failure in 1.27.1
<!--
1. If you have a question and not a bug/feature request please ask it at http://forum.serverless.com
2. Please check if an issue already exists so there are no duplicates
3. Check out and follow our Guidelines: https://github.com/serverless/serverless/blob/master/CONTRIBUTING.md
4. Fill out the whole template so we have a good overview on the issue
5. Do not remove any section of the template. If something is not applicable leave it empty but leave it in the Issue
6. Please follow the template, otherwise we'll have to ask you to update it
-->

# This is a Bug Report

## Description

In serverless 1.27.1 a nested variable expansion (e.g. `${file(serverless/default-vars.yml):greetings.${self:provider.awsProfile}}` ) that worked fine in 1.27.0 throws an error:

```
Serverless Warning --------------------------------------
 
  A valid file to satisfy the declaration 'file(serverless/default-vars.yml):greetings.self:provider.awsProfile' could not be found.
 
 
  Serverless Error ---------------------------------------
 
  Trying to populate non string value into a string for variable ${file(serverless/default-vars.yml):greetings.self:provider.awsProfile}. Please make sure the value of the property is a string.
```

I've reproduced an example that I can share:
[sls-error.zip](https://github.com/serverless/serverless/files/2076151/sls-error.zip)

Running `npm install` to install sls 1.27.1 and then `node_modules/.bin/sls package --stage live --profile qa --version 2 --region eu-west-1`.


## Additional Data




Issue discussion:
Having the same issue. This is related:
https://github.com/serverless/serverless/issues/4953
Hi @byarr. I can't reproduce the problem using the latest master https://github.com/serverless/serverless/commit/47816be84ae736f4827b860baa9689890d9bc133.
If the problem is fixed I think these issues can be closed:
https://github.com/serverless/serverless/issues/4953, https://github.com/serverless/serverless/issues/5044
I think the basic case is fixed, but not a slightly more complex but common one:

```yml
service: broken

provider:
  name: aws
  runtime: nodejs6.10
  environment:
    FOO: ${self:custom.config.test, self:provider.stage}

custom:
  config: ${file(./${opt:cfg}.yml)}

functions:
  hello:
    handler: handler.hello
```

We do this all the time where we have two configs, say `foo` and `bar`, and they may specify 'test' but we use some default 'test' as well, such as the stage name.

e.g.

foo.yml
```yml
test: 'override'
other: ignore
```
bar.yml
```yml
other: ignore
```

> sls package --cfg=foo

works

> sls package --cfg=bar

```bash
Serverless Warning --------------------------------------

  A valid undefined to satisfy the declaration 'deep:1.test' could not be found.

Serverless: Packaging service...
Serverless: Excluding development dependencies...

  Serverless Error ---------------------------------------

  Environment variable FOO must contain string

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           8.10.0
     Serverless Version:     1.27.0
```

It seems to start with 1.27.0 and  it has completely broken our ability to upgrade Serverless.

It looks like deep-variable messed up overwrite logic, because here:

https://github.com/serverless/serverless/blob/master/lib/classes/Variables.js#L456


`values` will be something like: `[ '${deep:0.test}', 'dev' ]`

and so "${deep:0.test}" satisfies the `validValue` logic. It's perhaps that the deep variable expansion isn't returning a promise that waits for the variable to be satisfied?
Thanks for doing the analysis of and providing materials to reproduce this @dougmoscrop.  Looking into it.

Solution patch:
diff --git a/lib/classes/Variables.js b/lib/classes/Variables.js
index 7cebde7aa7c..5de79cc82bc 100644
--- a/lib/classes/Variables.js
+++ b/lib/classes/Variables.js
@@ -453,8 +453,21 @@ class Variables {
       !(typeof value === 'object' && _.isEmpty(value))
     );
     return BbPromise.all(variableValues)
-      .then(values => // find and resolve first valid value, undefined if none
-        BbPromise.resolve(values.find(validValue)));
+      .then(values => {
+        let deepPropertyString = propertyString;
+        let deepProperties = 0;
+        values.forEach((value, index) => {
+          if (_.isString(value) && value.match(this.deepRefSyntax)) {
+            deepProperties += 1;
+            deepPropertyString = deepPropertyString.replace(
+              variableStrings[index],
+              this.cleanVariable(value));
+          }
+        });
+        return deepProperties > 0 ?
+          BbPromise.resolve(deepPropertyString) : // return deep variable replacement of original
+          BbPromise.resolve(values.find(validValue));// resolve first valid value, else undefined
+      });
   }
   /**
    * Given any variable string, return the value it should be populated with.


Test patch:
diff --git a/lib/classes/Variables.test.js b/lib/classes/Variables.test.js
index 1c8ec69eb24..08481b9d1a4 100644
--- a/lib/classes/Variables.test.js
+++ b/lib/classes/Variables.test.js
@@ -586,6 +586,20 @@ describe('Variables', () => {
         return serverless.variables.populateObject(service.custom)
           .should.become(expected);
       });
+      it('should handle overrides that are populated by unresolvable deep variables', () => {
+        service.custom = {
+          val0: 'foo',
+          val1: '${self:custom.val0}',
+          val2: '${self:custom.val1.notAnAttribute, "fallback"}',
+        };
+        const expected = {
+          val0: 'foo',
+          val1: 'foo',
+          val2: 'fallback',
+        };
+        return serverless.variables.populateObject(service.custom)
+          .should.become(expected);
+      });
       it('should handle deep variables regardless of custom variableSyntax', () => {
         service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._\\\'",\\-\\/\\(\\)]+?)}}';
         serverless.variables.loadVariableSyntax();