Repository: serverless/serverless (JavaScript) — Bug Fix

Issue:
A valid service attribute to satisfy the declaration 'self:service.name' could not be found
# This is a (Bug Report)

## Description

For bug reports:
* What went wrong?

I'm trying to use dynamic variables in `serverless.yml` and it fails.
Basically, I have the following in my `provider`:

```
  environment:
    SERVICE_NAME: ${self:service.name}
    DYNAMODB_TABLE: ${self:service.name}-test
```
The `SERVICE_NAME` is correctly resolved **IF** I comment the `DYNAMODB_TABLE: "${self:service.name}-test"`, otherwise it fails with the following stack trace.

If I use `${self:service.name}` alone it works, but if I add some string after, then it fails: `${self:service.name}-test`

* What did you expect should have happened?

Resolving variables correctly

* What stacktrace or error message from your provider did you see?

```bash
➜  simulator-feedback git:(master) ✗ sls print
Serverless: Load command run
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command emit
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command webpack
 
 Serverless Warning --------------------------------------
 
  A valid service attribute to satisfy the declaration 'self:service.name' could not be found.
 
 
 Serverless Warning --------------------------------------
 
  A valid service attribute to satisfy the declaration 'self:service.name' could not be found.
 
 
  Serverless Error ---------------------------------------
 
  Trying to populate non string value into a string for variable ${self:service.name}. Please make sure the value of the property is a string.
 
  Stack Trace --------------------------------------------
 
ServerlessError: Trying to populate non string value into a string for variable ${self:service.name}. Please make sure the value of the property is a string.
    at Variables.populateVariable (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/lib/classes/Variables.js:156:13)
    at pendingMatches.push.pendingMatch.then.matchedValue (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/lib/classes/Variables.js:124:21)
    at runCallback (timers.js:789:20)
    at tryOnImmediate (timers.js:751:5)
    at processImmediate [as _immediateCallback] (timers.js:722:5)
From previous event:
    at property.match.forEach (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/lib/classes/Variables.js:122:40)
    at Array.forEach (<anonymous>)
    at Variables.populateProperty (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/lib/classes/Variables.js:111:41)
    at forEachLeaf (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/lib/classes/Variables.js:85:14)
    at forEachLeaf (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/lib/classes/Variables.js:75:14)
    at addContext (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/lib/classes/Variables.js:62:42)
    at /Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/node_modules/lodash/lodash.js:13392:38
    at /Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/node_modules/lodash/lodash.js:4917:15
    at baseForOwn (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/node_modules/lodash/lodash.js:3002:24)
    at Function.mapValues (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/node_modules/lodash/lodash.js:13391:7)
    at forEachLeaf (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/lib/classes/Variables.js:73:37)
    at addContext (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/lib/classes/Variables.js:62:42)
    at /Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/node_modules/lodash/lodash.js:13392:38
    at /Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/node_modules/lodash/lodash.js:4917:15
    at baseForOwn (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/node_modules/lodash/lodash.js:3002:24)
    at Function.mapValues (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/node_modules/lodash/lodash.js:13391:7)
    at forEachLeaf (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/lib/classes/Variables.js:73:37)
    at addContext (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/lib/classes/Variables.js:62:42)
    at /Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/node_modules/lodash/lodash.js:13392:38
    at /Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/node_modules/lodash/lodash.js:4917:15
    at baseForOwn (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/node_modules/lodash/lodash.js:3002:24)
    at Function.mapValues (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/node_modules/lodash/lodash.js:13391:7)
    at forEachLeaf (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/lib/classes/Variables.js:73:37)
    at Variables.populateObject (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/lib/classes/Variables.js:79:5)
    at Variables.populateService (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/lib/classes/Variables.js:47:17)
    at Serverless.run (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/lib/Serverless.js:89:27)
    at serverless.init.then (/Users/vadorequest/.nvm/versions/node/v8.9.4/lib/node_modules/serverless/bin/serverless:42:50)
    at <anonymous>
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless
 
  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           8.9.4
     Serverless Version:     1.26.0

```

Similar or dependent issues:
* https://github.com/serverless/serverless/issues/3714

## Additional Data

* ***Operating System***: MacOS X



Discussion:
Additional information: This seems to be related to the `env` and particular `self:service` mostly, for instance:

- ✅ `Resource: "arn:aws:dynamodb:${self:provider.region}:035907498810:table/simulator-feedback-${self:provider.stage}"`
- ❗️ `Resource: "arn:aws:dynamodb:${self:provider.region}:035907498810:table/${self:service.name}-${self:provider.stage}"`
- ❗️ `Resource: "arn:aws:dynamodb:${self:provider.region}:035907498810:table/${env:SERVICE_NAME}-${self:provider.stage}"`

When using `self:provider`, it seems to work, but when using `env:` or `self:service.name`, it just fails. Maybe it's not possible to actually get the value from `self:service`? But I was following this official example so I assume it's possible but broken https://github.com/serverless/examples/blob/master/aws-node-rest-api-with-dynamodb/serverless.yml

I debug using the `sls print` command.
I've found the "name" value available at `${self:service}`. I agree that it's odd that it doesn't match the shape of the YAML, but it matches the shape of the value of the "service" field in ".serverless/serverless-state.json", so I suspect that's the input being used.

I'd guess that this is for backwards compatibility reasons? The presence of "serviceObject" with the field "name" seems like evidence of that.

Actually, I tried again and you're right. Both `SERVICE_NAME: ${self:service}` and `SERVICE_NAME: ${self:service.name}` work.

And both fail when used inside a string:

- `"${self:service}-${self:provider.stage}"`
- `"${self:service.name}-${self:provider.stage}"`

Also, when defining an `SERVICE_NAME` environment variable, those fail as well:

- `"${env:SERVICE_NAME}-${self:provider.stage}"`
- `"${self:provider.environment.SERVICE_NAME}-${self:provider.stage}"`

Failures seems to be related to the fact it's mixed up with string.
Could this get labeled/assigned? If you need more information, please let me know.
Can you try the same with the current master? I ask because #4713 just was merged, which cleans up variabe dependencies and also respects the use variables for region etc.
Yup, I can. I guess I just have to change my package.json to link to the github master branch instead of the npm package to do so?
Yes `"serverless": "github:serverless/serverless#master"`
Sumary
======

The PR seem to have fixed one use case when using ` "${self:service}-${self:provider.stage}"` (see Scenario 1). Possible backward breaking change added (See Scenario 2).

---

Running tests with the following setup:

`sls`: refers to my globally npm installed package `1.26.6`
`./node_modules/serverless/bin/serverless`: refers to my locally installed package, based on master branch at https://github.com/serverless/serverless/commit/39caddc404ecabf40ea3fb2aa51d31cbe8db7acd

# Scenario 1
`serverless.yml` file:

```
service:
  name: simulator-feedback
custom:
  test: "${self:service}-${self:provider.stage}"
```

## Test with sls 1.26.6 => Failure (expected)

```
(studylink-dev) ➜  simulator-feedback git:(master) ✗ sls print                                     
Serverless: Invoke print
 
  Serverless Error ---------------------------------------
 
  Trying to populate non string value into a string for variable ${self:service}. Please make sure the value of the property is a string.
 
  Stack Trace --------------------------------------------
 
ServerlessError: Trying to populate non string value into a string for variable ${self:service}. Please make sure the value of the property is a string.
    at Variables.populateVariable (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/Variables.js:156:13)
    at pendingMatches.push.pendingMatch.then.matchedValue (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/Variables.js:124:21)
From previous event:
    at property.match.forEach (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/Variables.js:122:40)
    at Array.forEach (native)
    at Variables.populateProperty (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/Variables.js:111:41)
    at forEachLeaf (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/Variables.js:85:14)
    at forEachLeaf (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/Variables.js:75:14)
    at addContext (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/Variables.js:62:42)
    at /Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/node_modules/lodash/lodash.js:13392:38
    at /Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/node_modules/lodash/lodash.js:4917:15
    at baseForOwn (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/node_modules/lodash/lodash.js:3002:24)
    at Function.mapValues (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/node_modules/lodash/lodash.js:13391:7)
    at forEachLeaf (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/Variables.js:73:37)
    at addContext (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/Variables.js:62:42)
    at /Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/node_modules/lodash/lodash.js:13392:38
    at /Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/node_modules/lodash/lodash.js:4917:15
    at baseForOwn (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/node_modules/lodash/lodash.js:3002:24)
    at Function.mapValues (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/node_modules/lodash/lodash.js:13391:7)
    at forEachLeaf (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/Variables.js:73:37)
    at Variables.populateObject (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/Variables.js:79:5)
    at getServerlessConfigFile.then (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/plugins/print/print.js:40:42)
From previous event:
    at Print.print (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/plugins/print/print.js:31:8)
From previous event:
    at Object.print:print [as hook] (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/plugins/print/print.js:22:10)
    at BbPromise.reduce (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/PluginManager.js:372:55)
From previous event:
    at PluginManager.invoke (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/PluginManager.js:372:22)
    at PluginManager.run (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/classes/PluginManager.js:403:17)
    at variables.populateService.then (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/Serverless.js:102:33)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)
From previous event:
    at Serverless.run (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/lib/Serverless.js:89:74)
    at serverless.init.then (/Users/vadorequest/.nvm/versions/node/v6.10.3/lib/node_modules/serverless/bin/serverless:42:50)
 
  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless
 
  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           6.10.3
     Serverless Version:     1.26.0
```

## Test with sls on master branch (success => 💯 )

```
./node_modules/serverless/bin/serverless print
custom:
  test: simulator-feedback-development

```

# Scenario 2
```
service:
  name: simulator-feedback
custom:
  test: "${self:service.name}-${self:provider.stage}"
```

**N.B:** added "self:service**.name**" compared to Scenario 1

## Test with sls (Failure, expected)
Same output as Scenario 1 sls

## Test with sls on master branch (failure => 👎  )

```
  A valid service attribute to satisfy the declaration 'self:service.name' could not be found.
 
 
  Serverless Error ---------------------------------------
 
  Trying to populate non string value into a string for variable ${self:service.name}. Please make sure the value of the property is a string.

```

> It looks like `${self:service.name}` is **invalid**, when ${self:service} is **valid**. This may lead to breaking changes.

# Scenario 3

I tried adding `test2: "${env:SERVICE_NAME}-${self:provider.stage}"` in `custom` and `provider`, both failed with the following message:

```
 Serverless Warning --------------------------------------
 
  A valid environment variable to satisfy the declaration 'env:SERVICE_NAME' could not be found.
 
 
  Serverless Error ---------------------------------------
 
  Trying to populate non string value into a string for variable ${env:SERVICE_NAME}. Please make sure the value of the property is a string.
 
  Stack Trace --------------------------------------------
 
ServerlessError: Trying to populate non string value into a string for variable ${env:SERVICE_NAME}. Please make sure the value of the property is a string.
    at Variables.populateVariable (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:361:13)
    at Variables.renderMatches (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:302:21)
    at BbPromise.all.then.results (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:322:29)
From previous event:
    at Variables.populateValue (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:322:8)
    at _.map.variable (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:206:24)
    at arrayMap (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/lodash/lodash.js:631:23)
    at Function.map (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/lodash/lodash.js:9546:14)
    at Variables.populateVariables (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:205:14)
    at Variables.populateObject (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:231:30)
    at prepopulateService.then (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:117:24)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)
From previous event:
    at Variables.populateService (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:117:8)
    at Serverless.run (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/Serverless.js:89:27)
    at serverless.init.then (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/bin/serverless:42:50)

```

I don't know how variables are intended to be used, but it looks like creating new variables based on existing environment variables just fail.

# Scenario 4

I tried adding `test2: "${self:custom.test}-${self:provider.stage}"` in `custom` and `provider`, both failed with the following message:

```
 Serverless Warning --------------------------------------
 
  A valid undefined to satisfy the declaration 'deep:0' could not be found.
 
 
  Serverless Error ---------------------------------------
 
  Trying to populate non string value into a string for variable ${deep:0}. Please make sure the value of the property is a string.
 
  Stack Trace --------------------------------------------
 
ServerlessError: Trying to populate non string value into a string for variable ${deep:0}. Please make sure the value of the property is a string.
    at Variables.populateVariable (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:361:13)
    at Variables.renderMatches (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:302:21)
    at BbPromise.all.then.results (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:322:29)
From previous event:
    at Variables.populateValue (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:322:8)
    at _.map.variable (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:206:24)
    at arrayMap (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/lodash/lodash.js:631:23)
    at Function.map (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/lodash/lodash.js:9546:14)
    at Variables.populateVariables (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:205:14)
    at Variables.populateObject (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:231:30)
    at assignProperties.then (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:234:14)
From previous event:
    at Variables.populateObject (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:233:8)
    at prepopulateService.then (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:117:24)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)
From previous event:
    at Variables.populateService (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/classes/Variables.js:117:8)
    at Serverless.run (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/lib/Serverless.js:89:27)
    at serverless.init.then (/Users/vadorequest/dev/student-loan-simulator-serverless/aws/simulator-feedback/node_modules/serverless/bin/serverless:42:50)

```



@Vadorequest Wow 💯 . Thanks for the very detailed report. That's really helpful.

@erikerikson Please have a look at the previous summary. Especially scenarios 3 and 4 are interesting, while scenario 3 might be an issue before the variable cleanup, I think scenario 4 is still something that is broken now.
Thank you very much for these cases.  Will definitely dive into them and fix.  There's a lot of untangling to do but we have a better position to do so from.

Resolving patch:
diff --git a/lib/classes/PromiseTracker.js b/lib/classes/PromiseTracker.js
index f7638535518..ea710720bbc 100644
--- a/lib/classes/PromiseTracker.js
+++ b/lib/classes/PromiseTracker.js
@@ -4,11 +4,15 @@ const logWarning = require('./Error').logWarning;
 
 class PromiseTracker {
   constructor() {
+    this.reset();
+  }
+  reset() {
     this.promiseList = [];
     this.promiseMap = {};
     this.startTime = Date.now();
   }
   start() {
+    this.reset();
     this.interval = setInterval(this.report.bind(this), 2500);
   }
   report() {
@@ -25,6 +29,7 @@ class PromiseTracker {
   }
   stop() {
     clearInterval(this.interval);
+    this.reset();
   }
   add(variable, prms, specifier) {
     const promise = prms;
diff --git a/lib/classes/Service.js b/lib/classes/Service.js
index 68991debd7e..6fdbbd4c3f1 100644
--- a/lib/classes/Service.js
+++ b/lib/classes/Service.js
@@ -8,6 +8,9 @@ const semver = require('semver');
 
 class Service {
   constructor(serverless, data) {
+    // #######################################################################
+    // ## KEEP SYNCHRONIZED WITH EQUIVALENT IN ~/lib/plugins/print/print.js ##
+    // #######################################################################
     this.serverless = serverless;
 
     // Default properties
@@ -107,6 +110,14 @@ class Service {
       throw new ServerlessError(`"provider" property is missing in ${serviceFilename}`);
     }
 
+    // #######################################################################
+    // ## KEEP SYNCHRONIZED WITH EQUIVALENT IN ~/lib/plugins/print/print.js ##
+    // #######################################################################
+    // #####################################################################
+    // ## KEEP SYNCHRONIZED WITH EQUIVALENT IN ~/lib/classes/Variables.js ##
+    // ##   there, see `getValueFromSelf`                                 ##
+    // ##   here, see below                                               ##
+    // #####################################################################
     if (!_.isObject(serverlessFile.provider)) {
       const providerName = serverlessFile.provider;
       serverlessFile.provider = {
diff --git a/lib/classes/Variables.js b/lib/classes/Variables.js
index 4c19f43c712..4926f31dec7 100644
--- a/lib/classes/Variables.js
+++ b/lib/classes/Variables.js
@@ -10,6 +10,33 @@ const fse = require('../utils/fs/fse');
 const logWarning = require('./Error').logWarning;
 const PromiseTracker = require('./PromiseTracker');
 
+/**
+ * Maintainer's notes:
+ *
+ * This is a tricky class to modify and maintain.  A few rules on how it works...
+ *
+ * 0. The population of a service consists of pre-population, followed by population.  Pre-
+ *   population consists of populating only those settings which are required for variable
+ *   resolution.  Current examples include region and stage as they must be resolved to a
+ *   concrete value before they can be used in the provider's `request` method (used by
+ *   `getValueFromCf`, `getValueFromS3`, and `getValueFromSsm`) to resolve the associated values.
+ *   Original issue: #4725
+ * 1. All variable populations occur in generations.  Each of these generations resolves each
+ *   present variable in the given object or property (i.e. terminal string properties and/or
+ *   property parts) once.  This is to say that recursive resolutions should not be made.  This is
+ *   because cyclic references are allowed [i.e. ${self:} and the like]) and doing so leads to
+ *   dependency and dead-locking issues.  This leads to a problem with deep value population (i.e.
+ *   populating ${self:foo.bar} when ${self:foo} has a value of {opt:bar}).  To pause that, one must
+ *   pause population, noting the continued depth to traverse.  This motivated "deep" variables.
+ *   Original issue #4687
+ * 2. The resolution of variables can get very confusing if the same object is used multiple times.
+ *   An example of this is the print command which *implicitly/invisibly* populates the
+ *   serverless.yml and then *explicitly/visibily* renders the same file again, without the
+ *   adornments that the framework's components make to the originally loaded service.  As a result,
+ *   it is important to reset this object for each use.
+ * 3. Note that despite some AWS code herein that this class is used in all plugins.  Obviously
+ *   users avoid the AWS-specific variable types when targetting other clouds.
+ */
 class Variables {
   constructor(serverless) {
     this.serverless = serverless;
@@ -17,83 +44,68 @@ class Variables {
     this.tracker = new PromiseTracker();
 
     this.deep = [];
-    this.deepRefSyntax = RegExp(/^(\${)?deep:\d+(\.[^}]+)*()}?$/);
+    this.deepRefSyntax = RegExp(/(\${)?deep:\d+(\.[^}]+)*()}?/);
     this.overwriteSyntax = RegExp(/\s*(?:,\s*)+/g);
     this.fileRefSyntax = RegExp(/^file\((~?[a-zA-Z0-9._\-/]+?)\)/g);
     this.envRefSyntax = RegExp(/^env:/g);
     this.optRefSyntax = RegExp(/^opt:/g);
     this.selfRefSyntax = RegExp(/^self:/g);
-    this.cfRefSyntax = RegExp(/^cf:/g);
-    this.s3RefSyntax = RegExp(/^s3:(.+?)\/(.+)$/);
     this.stringRefSyntax = RegExp(/(?:('|").*?\1)/g);
-    this.ssmRefSyntax = RegExp(/^ssm:([a-zA-Z0-9_.\-/]+)[~]?(true|false)?/);
+    this.cfRefSyntax = RegExp(/^(?:\${)?cf:/g);
+    this.s3RefSyntax = RegExp(/^(?:\${)?s3:(.+?)\/(.+)$/);
+    this.ssmRefSyntax = RegExp(/^(?:\${)?ssm:([a-zA-Z0-9_.\-/]+)[~]?(true|false)?/);
   }
 
   loadVariableSyntax() {
     this.variableSyntax = RegExp(this.service.provider.variableSyntax, 'g');
   }
+
+  initialCall(func) {
+    this.deep = [];
+    this.tracker.start();
+    return func().finally(() => {
+      this.tracker.stop();
+      this.deep = [];
+    });
+  }
+
   // #############
   // ## SERVICE ##
   // #############
-  prepopulateService() {
+  prepopulateObject(objectToPopulate) {
     const dependentServices = [
-      { name: 'CloudFormation', regex: this.cfRefSyntax },
-      { name: 'S3', regex: this.s3RefSyntax },
-      { name: 'SSM', regex: this.ssmRefSyntax },
+      { name: 'CloudFormation', method: 'getValueFromCf', original: this.getValueFromCf },
+      { name: 'S3', method: 'getValueFromS3', original: this.getValueFromS3 },
+      { name: 'SSM', method: 'getValueFromSsm', original: this.getValueFromSsm },
     ];
     const dependencyMessage = (configName, configValue, serviceName) =>
-      `Variable Failure: value ${
-        configName
-        } set to '${
-        configValue
-        }' references ${
-        serviceName
-        } which requires a ${
-        configName
-        } value for use.`;
-    const getVariableParts = (variableString) => {
-      const matches = this.getMatches(variableString);
-      return matches.reduce(
-        (accumulation, current) =>
-          accumulation.concat(this.splitByComma(current.variable)),
-        []);
-    };
-    const serviceMatch = variablePart =>
-      dependentServices.find((service) => {
-        let variable = variablePart;
-        if (variable.match(this.deepRefSyntax)) {
-          variable = this.getVariableFromDeep(variablePart);
-        }
-        return variable.match(service.regex);
+      `Variable Failure: value ${configName} set to '${configValue}' references ${
+        serviceName} which requires a ${configName} value for use.`;
+    // replace and then restore the methods for obtaining values from dependent services.  the
+    // replacement naturally rejects dependencies on these services that occur during prepopulation.
+    // prepopulation is, of course, the process of obtaining the required configuration for using
+    // these services.
+    dependentServices.forEach((dependentService) => { // knock out
+      this[dependentService.method] = (variableString) => BbPromise.reject(
+        dependencyMessage(objectToPopulate.name, variableString, dependentService.name));
+    });
+    return this.populateValue(objectToPopulate.value, true) // populate
+      .then(populated => _.assign(objectToPopulate, { populated }))
+      .finally(() => {
+        dependentServices.forEach((dependentService) => { // restore
+          this[dependentService.method] = dependentService.original;
+        });
       });
-    const getUntilValid = (config) => {
-      const parts = getVariableParts(config.value);
-      const service = parts.reduce(
-        (accumulation, part) => accumulation || serviceMatch(part),
-        undefined);
-      if (service) {
-        const msg = dependencyMessage(config.name, config.value, service.name);
-        return BbPromise.reject(new this.serverless.classes.Error(msg));
-      }
-      return this.populateValue(config.value, false)
-        .then(populated => (
-          populated.match(this.variableSyntax) ?
-            getUntilValid(_.assign(config, { value: populated })) :
-            _.assign({}, config, { populated })
-        ));
-    };
-
+  }
+  prepopulateService() {
     const provider = this.serverless.getProvider('aws');
     if (provider) {
       const requiredConfig = [
         _.assign({ name: 'region' }, provider.getRegionSourceValue()),
         _.assign({ name: 'stage' }, provider.getStageSourceValue()),
       ];
-      const configToPopulate = requiredConfig.filter(config =>
-        !_.isUndefined(config.value) &&
-        (_.isString(config.value) && config.value.match(this.variableSyntax)));
-      const configPromises = configToPopulate.map(getUntilValid);
-      return this.assignProperties(provider, configPromises);
+      const prepopulations = requiredConfig.map(this.prepopulateObject.bind(this));
+      return this.assignProperties(provider, prepopulations);
     }
     return BbPromise.resolve();
   }
@@ -104,6 +116,9 @@ class Variables {
    * @returns {Promise.<TResult>|*} A promise resolving to the populated service.
    */
   populateService(processedOptions) {
+    // #######################################################################
+    // ## KEEP SYNCHRONIZED WITH EQUIVALENT IN ~/lib/plugins/print/print.js ##
+    // #######################################################################
     this.options = processedOptions || {};
 
     this.loadVariableSyntax();
@@ -112,16 +127,15 @@ class Variables {
     // remove
     this.service.provider.variableSyntax = undefined; // otherwise matches itself
     this.service.serverless = undefined;
-    this.tracker.start();
-    return this.prepopulateService()
-      .then(() => this.populateObject(this.service))
-      .finally(() => {
-        // restore
-        this.tracker.stop();
-        this.service.serverless = this.serverless;
-        this.service.provider.variableSyntax = variableSyntaxProperty;
-      })
-      .then(() => this.service);
+    return this.initialCall(() =>
+      this.prepopulateService()
+        .then(() => this.populateObjectImpl(this.service)
+          .finally(() => {
+            // restore
+            this.service.serverless = this.serverless;
+            this.service.provider.variableSyntax = variableSyntaxProperty;
+          }))
+        .then(() => this.service));
   }
   // ############
   // ## OBJECT ##
@@ -227,12 +241,16 @@ class Variables {
    * @returns {Promise.<TResult>|*} A promise resolving to the in-place populated object.
    */
   populateObject(objectToPopulate) {
+    return this.initialCall(() => this.populateObjectImpl(objectToPopulate));
+  }
+  populateObjectImpl(objectToPopulate) {
     const leaves = this.getProperties(objectToPopulate, true, objectToPopulate);
     const populations = this.populateVariables(leaves);
+    if (populations.length === 0) {
+      return BbPromise.resolve(objectToPopulate);
+    }
     return this.assignProperties(objectToPopulate, populations)
-      .then(() => (populations.length ?
-        this.populateObject(objectToPopulate) :
-        objectToPopulate));
+      .then(() => this.populateObjectImpl(objectToPopulate));
   }
   // ##############
   // ## PROPERTY ##
@@ -279,14 +297,8 @@ class Variables {
    * @param {MatchResult[]} matches The matches to populate
    * @returns {Promise[]} Promises for the eventual populated values of the given matches
    */
-  populateMatches(matches) {
-    return _.map(matches, (match) => {
-      const parts = this.splitByComma(match.variable);
-      if (parts.length > 1) {
-        return this.overwrite(parts, match.match);
-      }
-      return this.getValueFromSource(parts[0], match.match);
-    });
+  populateMatches(matches, property) {
+    return _.map(matches, (match) => this.splitAndGet(match.variable, property));
   }
   /**
    * Render the given matches and their associated results to the given value
@@ -312,12 +324,12 @@ class Variables {
    * is true
    */
   populateValue(valueToPopulate, root) {
-    const property = _.cloneDeep(valueToPopulate);
+    const property = valueToPopulate;
     const matches = this.getMatches(property);
     if (!_.isArray(matches)) {
       return BbPromise.resolve(property);
     }
-    const populations = this.populateMatches(matches);
+    const populations = this.populateMatches(matches, valueToPopulate);
     return BbPromise.all(populations)
       .then(results => this.renderMatches(property, matches, results))
       .then((result) => {
@@ -333,7 +345,22 @@ class Variables {
    * @returns {Promise.<TResult>|*} A promise resolving to the populated result.
    */
   populateProperty(propertyToPopulate) {
-    return this.populateValue(propertyToPopulate, true);
+    return this.initialCall(() => this.populateValue(propertyToPopulate, true));
+  }
+
+  /**
+   * Split the cleaned variable string containing one or more comma delimited variables and get a
+   * final value for the entirety of the string
+   * @param varible The variable string to split and get a final value for
+   * @param property The original property string the given variable was extracted from
+   * @returns {Promise} A promise resolving to the final value of the given variable
+   */
+  splitAndGet(variable, property) {
+    const parts = this.splitByComma(variable);
+    if (parts.length > 1) {
+      return this.overwrite(parts, property);
+    }
+    return this.getValueFromSource(parts[0], property);
   }
   /**
    * Populate a given property, given the matched string to replace and the value to replace the
@@ -495,9 +522,22 @@ class Variables {
   }
 
   getValueFromSelf(variableString) {
+    let variable = variableString;
+    // ###################################################################
+    // ## KEEP SYNCHRONIZED WITH EQUIVALENT IN ~/lib/classes/Service.js ##
+    // ##   there, see `loadServiceFileParam`                           ##
+    // ###################################################################
+    // The loaded service is altered during load in ~/lib/classes/Service (see loadServiceFileParam)
+    // Account for these so that user's reference to their file populate properly
+    if (variable === 'self:service.name') {
+      variable = 'self:service';
+    }
+    if (variable === 'self:provider') {
+      variable = 'self:provider.name';
+    }
     const valueToPopulate = this.service;
-    const deepProperties = variableString.split(':')[1].split('.');
-    return this.getDeepValue(deepProperties, valueToPopulate);
+    const deepProperties = variable.split(':')[1].split('.').filter(property => property);
+    return this.getDeeperValue(deepProperties, valueToPopulate);
   }
 
   getValueFromFile(variableString) {
@@ -552,7 +592,7 @@ class Variables {
         let deepProperties = variableString.replace(matchedFileRefString, '');
         deepProperties = deepProperties.slice(1).split('.');
         deepProperties.splice(0, 1);
-        return this.getDeepValue(deepProperties, valueToPopulateResolved)
+        return this.getDeeperValue(deepProperties, valueToPopulateResolved)
           .then((deepValueToPopulateResolved) => {
             if (typeof deepValueToPopulateResolved === 'undefined') {
               const errorMessage = [
@@ -582,7 +622,7 @@ class Variables {
           return BbPromise.reject(new this.serverless.classes.Error(errorMessage));
         }
         deepProperties = deepProperties.slice(1).split('.');
-        return this.getDeepValue(deepProperties, valueToPopulate);
+        return this.getDeeperValue(deepProperties, valueToPopulate);
       }
     }
     return BbPromise.resolve(valueToPopulate);
@@ -666,42 +706,60 @@ class Variables {
     const deepPrefixReplace = RegExp(/(?:^deep:)\d+\.?/g);
     const variable = this.getVariableFromDeep(variableString);
     const deepRef = variableString.replace(deepPrefixReplace, '');
-    const sourceString = `\${deep:\${${variable}}${deepRef.length ? `.${deepRef}` : ''}}`;
-    return this.getValueFromSource(variable, sourceString);
-  }
-
-  getDeepValue(deepProperties, valueToPopulate) {
-    return BbPromise.reduce(deepProperties, (computedValueToPopulateParam, subProperty) => {
-      let computedValueToPopulate = computedValueToPopulateParam;
-      if ( // in build deep variable mode
-        _.isString(computedValueToPopulate) &&
-        computedValueToPopulate.match(this.deepRefSyntax)
-      ) {
-        if (subProperty !== '') {
-          computedValueToPopulate = `${
-            computedValueToPopulate.slice(0, computedValueToPopulate.length - 1)
-          }.${
-            subProperty
-          }}`;
+    let ret = this.populateValue(variable);
+    if (deepRef.length) { // if there is a deep reference remaining
+      ret = ret.then((result) => {
+        if (_.isString(result) && result.match(this.variableSyntax)) {
+          const deepVariable = this.makeDeepVariable(result);
+          return BbPromise.resolve(this.appendDeepVariable(deepVariable, deepRef));
         }
-        return BbPromise.resolve(computedValueToPopulate);
-      } else if (typeof computedValueToPopulate === 'undefined') { // in get deep value mode
-        computedValueToPopulate = {};
-      } else if (subProperty !== '' || '' in computedValueToPopulate) {
-        computedValueToPopulate = computedValueToPopulate[subProperty];
-      }
-      if (
-        typeof computedValueToPopulate === 'string' &&
-        computedValueToPopulate.match(this.variableSyntax)
-      ) {
-        const computedVariable = this.cleanVariable(computedValueToPopulate);
-        let index = this.deep.findIndex((item) => computedVariable === item);
-        if (index < 0) {
-          index = this.deep.push(computedVariable) - 1;
+        return this.getDeeperValue(deepRef.split('.'), result);
+      });
+    }
+    return ret;
+  }
+
+  makeDeepVariable(variable) {
+    let index = this.deep.findIndex((item) => variable === item);
+    if (index < 0) {
+      index = this.deep.push(variable) - 1;
+    }
+    return `\${deep:${index}}`;
+  }
+  appendDeepVariable(variable, subProperty) {
+    return `${variable.slice(0, variable.length - 1)}.${subProperty}}`;
+  }
+
+  /**
+   * Get a value that is within the given valueToPopulate.  The deepProperties specify what value
+   * to retrieve from the given valueToPopulate.  The trouble is that anywhere along this chain a
+   * variable can be discovered.  If this occurs, to avoid cyclic dependencies, the resolution of
+   * the deep value from the given valueToPopulate must be halted.  The discovered variable is thus
+   * set aside into a "deep variable" (see makeDeepVariable).  The indexing into the given
+   * valueToPopulate is then resolved with a replacement ${deep:${index}.${remaining.properties}}
+   * variable (e.g. ${deep:1.foo}).  This pauses the population for continuation during the next
+   * generation of evaluation (see getValueFromDeep)
+   * @param deepProperties The "path" of properties to follow in obtaining the deeper value
+   * @param valueToPopulate The value from which to obtain the deeper value
+   * @returns {Promise} A promise resolving to the deeper value or to a `deep` variable that
+   * will later resolve to the deeper value
+   */
+  getDeeperValue(deepProperties, valueToPopulate) {
+    return BbPromise.reduce(deepProperties, (reducedValueParam, subProperty) => {
+      let reducedValue = reducedValueParam;
+      if (_.isString(reducedValue) && reducedValue.match(this.deepRefSyntax)) { // build mode
+        reducedValue = this.appendDeepVariable(reducedValue, subProperty);
+      } else { // get mode
+        if (typeof reducedValue === 'undefined') {
+          reducedValue = {};
+        } else if (subProperty !== '' || '' in reducedValue) {
+          reducedValue = reducedValue[subProperty];
+        }
+        if (typeof reducedValue === 'string' && reducedValue.match(this.variableSyntax)) {
+          reducedValue = this.makeDeepVariable(reducedValue);
         }
-        return BbPromise.resolve(`\${deep:${index}}`);
       }
-      return BbPromise.resolve(computedValueToPopulate);
+      return BbPromise.resolve(reducedValue);
     }, valueToPopulate);
   }
 
diff --git a/lib/plugins/print/print.js b/lib/plugins/print/print.js
index b48f26e36f8..80cf59a53f7 100644
--- a/lib/plugins/print/print.js
+++ b/lib/plugins/print/print.js
@@ -1,5 +1,6 @@
 'use strict';
 
+const _ = require('lodash');
 const BbPromise = require('bluebird');
 const getServerlessConfigFile = require('../../utils/getServerlessConfigFile');
 const jc = require('json-cycle');
@@ -8,6 +9,7 @@ const YAML = require('js-yaml');
 class Print {
   constructor(serverless) {
     this.serverless = serverless;
+    this.cache = {};
 
     this.commands = {
       print: {
@@ -23,29 +25,76 @@ class Print {
     };
   }
 
+  adorn(svc) {
+    const service = svc;
+    // service object treatment
+    this.cache.serviceService = service.service;
+    if (_.isObject(this.cache.serviceService)) {
+      service.service = service.service.name;
+      service.serviceObject = this.cache.serviceService;
+    }
+    // provider treatment and defaults
+    this.cache.serviceProvider = service.provider;
+    if (_.isString(this.cache.serviceProvider)) {
+      service.provider = { name: this.cache.serviceProvider };
+    }
+    service.provider = _.merge({
+      stage: 'dev',
+      region: 'us-east-1',
+      variableSyntax: '\\${([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}',
+    }, service.provider);
+  }
+  strip(svc) {
+    const service = svc;
+    if (_.isObject(this.cache.serviceService)) {
+      service.service = this.cache.serviceService;
+      delete service.serviceObject;
+    }
+    if (_.isString(this.cache.serviceProvider)) {
+      service.provider = this.cache.serviceProvider;
+    } else { // is object
+      if (!this.cache.serviceProvider.stage) {
+        delete service.provider.stage;
+      }
+      if (!this.cache.serviceProvider.region) {
+        delete service.provider.region;
+      }
+      if (this.cache.serviceProvider.variableSyntax) {
+        service.provider.variableSyntax = this.cache.serviceProvider.variableSyntax;
+      }
+    }
+  }
   print() {
-    let variableSyntax;
-    this.serverless.variables.options = this.serverless.processedInput.options;
-    this.serverless.variables.loadVariableSyntax();
+    // #####################################################################
+    // ## KEEP SYNCHRONIZED WITH EQUIVALENT IN ~/lib/classes/Variables.js ##
+    // ##   there, see `populateService`                                  ##
+    // ##   here, see below                                               ##
+    // #####################################################################
+    // ###################################################################
+    // ## KEEP SYNCHRONIZED WITH EQUIVALENT IN ~/lib/classes/Service.js ##
+    // ##   there, see `constructor` and `loadServiceFileParam`         ##
+    // ##   here, see `strip` and `adorn`                               ##
+    // ###################################################################
+    // The *already loaded* Service (i.e. serverless.yml) is adorned with extras for use by the
+    // framework and throughout its codebase.  We could try using the populated service but that
+    // would require playing whack-a-mole on issues as changes are made to the service anywhere in
+    // the codebase.  Avoiding that, this method must read the serverless.yml file itself, adorn it
+    // as the Service class would and then populate it, reversing the adornments thereafter in
+    // preparation for printing the service for the user.
     return getServerlessConfigFile(process.cwd())
-      .then((data) => {
-        const conf = data;
-        // Need to delete variableSyntax to avoid potential matching errors
-        if (conf.provider.variableSyntax) {
-          variableSyntax = conf.provider.variableSyntax;
-          delete conf.provider.variableSyntax;
-        }
-        this.serverless.variables.service = conf;
-        this.serverless.variables.cache = {};
-        return this.serverless.variables.populateObject(conf);
-      })
-      .then((data) => {
-        const conf = data;
-        if (variableSyntax !== undefined) {
-          conf.provider.variableSyntax = variableSyntax;
-        }
-        const out = JSON.parse(jc.stringify(conf));
-        this.serverless.cli.consoleLog(YAML.dump(out, { noRefs: true }));
+      .then((svc) => {
+        const service = svc;
+        this.adorn(service);
+        // Need to delete variableSyntax to avoid self-matching errors
+        this.serverless.variables.loadVariableSyntax();
+        delete service.provider.variableSyntax; // cached by adorn, restored by strip
+        this.serverless.variables.service = service;
+        return this.serverless.variables.populateObject(service)
+          .then((populated) => {
+            this.strip(populated);
+            const out = JSON.parse(jc.stringify(populated));
+            this.serverless.cli.consoleLog(YAML.dump(out, { noRefs: true }));
+          });
       });
   }
 


Test patch:
diff --git a/lib/classes/Variables.test.js b/lib/classes/Variables.test.js
index 6520f7c2bf0..8962814e371 100644
--- a/lib/classes/Variables.test.js
+++ b/lib/classes/Variables.test.js
@@ -21,11 +21,11 @@ const Variables = require('../../lib/classes/Variables');
 
 BbPromise.longStackTraces(true);
 
-chai.should();
-
 chai.use(require('chai-as-promised'));
 chai.use(require('sinon-chai'));
 
+chai.should();
+
 const expect = chai.expect;
 
 
@@ -55,32 +55,73 @@ describe('Variables', () => {
   });
 
   describe('#populateService()', () => {
-    it('should call loadVariableSyntax and then populateProperty', () => {
-      const loadVariableSyntaxStub = sinon.stub(serverless.variables, 'loadVariableSyntax')
-        .returns();
-      const populateObjectStub = sinon.stub(serverless.variables, 'populateObject')
-        .returns(BbPromise.resolve());
-      return expect(serverless.variables.populateService()).to.be.fulfilled
-        .then(() => {
-          expect(loadVariableSyntaxStub).to.be.calledOnce;
-          expect(populateObjectStub).to.be.calledOnce;
-          expect(loadVariableSyntaxStub).to.be.calledBefore(populateObjectStub);
-        })
-        .finally(() => {
-          populateObjectStub.restore();
-          loadVariableSyntaxStub.restore();
-        });
-    });
-    it('should remove problematic attributes bofore calling populateObject with the service',
+    it('should remove problematic attributes bofore calling populateObjectImpl with the service',
       () => {
-        const populateObjectStub = sinon.stub(serverless.variables, 'populateObject', (val) => {
+        const prepopulateServiceStub = sinon.stub(serverless.variables, 'prepopulateService')
+          .returns(BbPromise.resolve());
+        const populateObjectStub = sinon.stub(serverless.variables, 'populateObjectImpl', (val) => {
           expect(val).to.equal(serverless.service);
           expect(val.provider.variableSyntax).to.be.undefined;
           expect(val.serverless).to.be.undefined;
           return BbPromise.resolve();
         });
-        return expect(serverless.variables.populateService()).to.be.fulfilled
-          .then().finally(() => populateObjectStub.restore());
+        return serverless.variables.populateService().should.be.fulfilled
+          .then().finally(() => {
+            prepopulateServiceStub.restore();
+            populateObjectStub.restore();
+          });
+      });
+    it('should clear caches and remaining state *before* [pre]populating service',
+      () => {
+        const prepopulateServiceStub = sinon.stub(serverless.variables, 'prepopulateService',
+          (val) => {
+            expect(serverless.variables.deep).to.eql([]);
+            expect(serverless.variables.tracker.getAll()).to.eql([]);
+            return BbPromise.resolve(val);
+          });
+        const populateObjectStub = sinon.stub(serverless.variables, 'populateObjectImpl',
+          (val) => {
+            expect(serverless.variables.deep).to.eql([]);
+            expect(serverless.variables.tracker.getAll()).to.eql([]);
+            return BbPromise.resolve(val);
+          });
+        serverless.variables.deep.push('${foo:}');
+        const prms = BbPromise.resolve('foo');
+        serverless.variables.tracker.add('foo:', prms, '${foo:}');
+        prms.state = 'resolved';
+        return serverless.variables.populateService().should.be.fulfilled
+          .then().finally(() => {
+            prepopulateServiceStub.restore();
+            populateObjectStub.restore();
+          });
+      });
+    it('should clear caches and remaining *after* [pre]populating service',
+      () => {
+        const prepopulateServiceStub = sinon.stub(serverless.variables, 'prepopulateService',
+          (val) => {
+            serverless.variables.deep.push('${foo:}');
+            const promise = BbPromise.resolve(val);
+            serverless.variables.tracker.add('foo:', promise, '${foo:}');
+            promise.state = 'resolved';
+            return BbPromise.resolve();
+          });
+        const populateObjectStub = sinon.stub(serverless.variables, 'populateObjectImpl',
+          (val) => {
+            serverless.variables.deep.push('${bar:}');
+            const promise = BbPromise.resolve(val);
+            serverless.variables.tracker.add('bar:', promise, '${bar:}');
+            promise.state = 'resolved';
+            return BbPromise.resolve();
+          });
+        return serverless.variables.populateService().should.be.fulfilled
+          .then(() => {
+            expect(serverless.variables.deep).to.eql([]);
+            expect(serverless.variables.tracker.getAll()).to.eql([]);
+          })
+          .finally(() => {
+            prepopulateServiceStub.restore();
+            populateObjectStub.restore();
+          });
       });
   });
   describe('#prepopulateService', () => {
@@ -90,17 +131,17 @@ describe('Variables', () => {
     // variable syntax is loaded, and that the service object is cleaned up.  Just use
     // populateService to do that work.
     let awsProvider;
-    let populateObjectStub;
+    let populateObjectImplStub;
     let requestStub; // just in case... don't want to actually call...
     beforeEach(() => {
       awsProvider = new AwsProvider(serverless, {});
-      populateObjectStub = sinon.stub(serverless.variables, 'populateObject', () =>
-        BbPromise.resolve());
+      populateObjectImplStub = sinon.stub(serverless.variables, 'populateObjectImpl');
+      populateObjectImplStub.withArgs(serverless.variables.service).returns(BbPromise.resolve());
       requestStub = sinon.stub(awsProvider, 'request', () =>
         BbPromise.reject(new Error('unexpected')));
     });
     afterEach(() => {
-      populateObjectStub.restore();
+      populateObjectImplStub.restore();
       requestStub.restore();
     });
     const prepopulatedProperties = [
@@ -130,16 +171,16 @@ describe('Variables', () => {
             return serverless.variables.populateService()
               .should.be.rejectedWith('Variable Failure');
           });
+          it(`should reject recursively dependent ${config.name} service dependencies`, () => {
+            serverless.variables.service.custom = {
+              settings: config.value,
+            };
+            awsProvider.options.region = '${self:custom.settings.region}';
+            return serverless.variables.populateService()
+              .should.be.rejectedWith('Variable Failure');
+          });
         });
       });
-      it('should reject recursively dependent service dependencies', () => {
-        serverless.variables.service.custom = {
-          settings: '${s3:bucket/key}',
-        };
-        awsProvider.options.region = '${self:custom.settings.region}';
-        return serverless.variables.populateService()
-          .should.be.rejectedWith('Variable Failure');
-      });
     });
   });
 
@@ -390,6 +431,136 @@ describe('Variables', () => {
           expect(result).to.eql(expected);
         })).to.be.fulfilled;
       });
+      // see https://github.com/serverless/serverless/pull/4713#issuecomment-366975172
+      it('should handle deep references into deep variables', () => {
+        service.provider.stage = 'dev';
+        service.custom = {
+          stage: '${env:stage, self:provider.stage}',
+          secrets: '${self:custom.${self:custom.stage}}',
+          dev: {
+            SECRET: 'secret',
+          },
+          environment: {
+            SECRET: '${self:custom.secrets.SECRET}',
+          },
+        };
+        const expected = {
+          stage: 'dev',
+          secrets: {
+            SECRET: 'secret',
+          },
+          dev: {
+            SECRET: 'secret',
+          },
+          environment: {
+            SECRET: 'secret',
+          },
+        };
+        return serverless.variables.populateObject(service.custom)
+          .should.become(expected);
+      });
+      it('should handle deep variables that reference overrides', () => {
+        service.custom = {
+          val1: '${self:not.a.value, "bar"}',
+          val2: '${self:custom.val1}',
+        };
+        const expected = {
+          val1: 'bar',
+          val2: 'bar',
+        };
+        return serverless.variables.populateObject(service.custom)
+          .should.become(expected);
+      });
+      it('should handle deep references into deep variables', () => {
+        service.custom = {
+          val0: {
+            foo: 'bar',
+          },
+          val1: '${self:custom.val0}',
+          val2: '${self:custom.val1.foo}',
+        };
+        const expected = {
+          val0: {
+            foo: 'bar',
+          },
+          val1: {
+            foo: 'bar',
+          },
+          val2: 'bar',
+        };
+        return serverless.variables.populateObject(service.custom)
+          .should.become(expected);
+      });
+      it('should handle deep variables that reference overrides', () => {
+        service.custom = {
+          val1: '${self:not.a.value, "bar"}',
+          val2: 'foo${self:custom.val1}',
+        };
+        const expected = {
+          val1: 'bar',
+          val2: 'foobar',
+        };
+        return serverless.variables.populateObject(service.custom)
+          .should.become(expected);
+      });
+      it('should handle referenced deep variables that reference overrides', () => {
+        service.custom = {
+          val1: '${self:not.a.value, "bar"}',
+          val2: '${self:custom.val1}',
+          val3: '${self:custom.val2}',
+        };
+        const expected = {
+          val1: 'bar',
+          val2: 'bar',
+          val3: 'bar',
+        };
+        return serverless.variables.populateObject(service.custom)
+          .should.become(expected);
+      });
+      it('should handle partial referenced deep variables that reference overrides', () => {
+        service.custom = {
+          val1: '${self:not.a.value, "bar"}',
+          val2: '${self:custom.val1}',
+          val3: 'foo${self:custom.val2}',
+        };
+        const expected = {
+          val1: 'bar',
+          val2: 'bar',
+          val3: 'foobar',
+        };
+        return serverless.variables.populateObject(service.custom)
+          .should.become(expected);
+      });
+      it('should handle referenced contained deep variables that reference overrides', () => {
+        service.custom = {
+          val1: '${self:not.a.value, "bar"}',
+          val2: 'foo${self:custom.val1}',
+          val3: '${self:custom.val2}',
+        };
+        const expected = {
+          val1: 'bar',
+          val2: 'foobar',
+          val3: 'foobar',
+        };
+        return serverless.variables.populateObject(service.custom)
+          .should.become(expected);
+      });
+      it('should handle multiple referenced contained deep variables referencing overrides', () => {
+        service.custom = {
+          val0: '${self:not.a.value, "foo"}',
+          val1: '${self:not.a.value, "bar"}',
+          val2: '${self:custom.val0}:${self:custom.val1}',
+          val3: '${self:custom.val2}',
+        };
+        const expected = {
+          val0: 'foo',
+          val1: 'bar',
+          val2: 'foo:bar',
+          val3: 'foo:bar',
+        };
+        return serverless.variables.populateObject(service.custom)
+          .should.become(expected);
+      });
       describe('file reading cases', () => {
         let tmpDirPath;
         beforeEach(() => {
@@ -915,22 +1086,39 @@ module.exports = {
   });
 
   describe('#getValueFromSelf()', () => {
+    beforeEach(() => {
+      serverless.service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}}';
+      serverless.variables.loadVariableSyntax();
+      delete serverless.service.provider.variableSyntax;
+    });
     it('should get variable from self serverless.yml file', () => {
       serverless.variables.service = {
         service: 'testService',
         provider: serverless.service.provider,
       };
-      serverless.variables.loadVariableSyntax();
       return serverless.variables.getValueFromSelf('self:service').should.become('testService');
     });
-
+    it('should redirect ${self:service.name} to ${self:service}', () => {
+      serverless.variables.service = {
+        service: 'testService',
+        provider: serverless.service.provider,
+      };
+      return serverless.variables.getValueFromSelf('self:service.name')
+        .should.become('testService');
+    });
+    it('should redirect ${self:provider} to ${self:provider.name}', () => {
+      serverless.variables.service = {
+        service: 'testService',
+        provider: { name: 'aws' },
+      };
+      return serverless.variables.getValueFromSelf('self:provider').should.become('aws');
+    });
     it('should handle self-references to the root of the serverless.yml file', () => {
       serverless.variables.service = {
         service: 'testService',
         provider: 'testProvider',
         defaults: serverless.service.defaults,
       };
-      serverless.variables.loadVariableSyntax();
       return serverless.variables.getValueFromSelf('self:')
         .should.eventually.equal(serverless.variables.service);
     });
@@ -1353,7 +1541,7 @@ module.exports = {
     });
   });
 
-  describe('#getDeepValue()', () => {
+  describe('#getDeeperValue()', () => {
     it('should get deep values', () => {
       const valueToPopulateMock = {
         service: 'testService',
@@ -1364,7 +1552,7 @@ module.exports = {
         },
       };
       serverless.variables.loadVariableSyntax();
-      return serverless.variables.getDeepValue(['custom', 'subProperty', 'deep'],
+      return serverless.variables.getDeeperValue(['custom', 'subProperty', 'deep'],
         valueToPopulateMock).should.become('deepValue');
     });
     it('should not throw error if referencing invalid properties', () => {
@@ -1375,7 +1563,7 @@ module.exports = {
         },
       };
       serverless.variables.loadVariableSyntax();
-      return serverless.variables.getDeepValue(['custom', 'subProperty', 'deep', 'deeper'],
+      return serverless.variables.getDeeperValue(['custom', 'subProperty', 'deep', 'deeper'],
         valueToPopulateMock).should.eventually.deep.equal({});
     });
     it('should return a simple deep variable when final deep value is variable', () => {
@@ -1390,7 +1578,7 @@ module.exports = {
         provider: serverless.service.provider,
       };
       serverless.variables.loadVariableSyntax();
-      return serverless.variables.getDeepValue(
+      return serverless.variables.getDeeperValue(
         ['custom', 'subProperty', 'deep'],
         serverless.variables.service
       ).should.become('${deep:0}');
@@ -1404,7 +1592,7 @@ module.exports = {
         provider: serverless.service.provider,
       };
       serverless.variables.loadVariableSyntax();
-      return serverless.variables.getDeepValue(
+      return serverless.variables.getDeeperValue(
         ['custom', 'anotherVar', 'veryDeep'],
         serverless.variables.service)
         .should.become('${deep:0.veryDeep}');
diff --git a/lib/plugins/print/print.test.js b/lib/plugins/print/print.test.js
index ec4dc1950e0..7ed46582ef0 100644
--- a/lib/plugins/print/print.test.js
+++ b/lib/plugins/print/print.test.js
@@ -19,9 +19,9 @@ describe('Print', () => {
       '../../utils/getServerlessConfigFile': getServerlessConfigFileStub,
     });
     serverless = new Serverless();
-    serverless.processedInput = {
-      commands: ['print'],
-      options: { stage: undefined, region: undefined },
+    serverless.variables.options = {
+      stage: 'dev',
+      region: 'us-east-1',
     };
     serverless.cli = new CLI(serverless);
     print = new PrintPlugin(serverless);
@@ -48,7 +48,25 @@ describe('Print', () => {
 
       expect(getServerlessConfigFileStub.calledOnce).to.equal(true);
       expect(print.serverless.cli.consoleLog.called).to.be.equal(true);
-      expect(message).to.have.string(YAML.dump(conf));
+      expect(YAML.load(message)).to.eql(conf);
+    });
+  });
+
+  it('should print special service object and provider string configs', () => {
+    const conf = {
+      service: {
+        name: 'my-service',
+      },
+      provider: 'aws',
+    };
+    getServerlessConfigFileStub.resolves(conf);
+
+    return print.print().then(() => {
+      const message = print.serverless.cli.consoleLog.args.join();
+
+      expect(getServerlessConfigFileStub.calledOnce).to.equal(true);
+      expect(print.serverless.cli.consoleLog.called).to.be.equal(true);
+      expect(YAML.load(message)).to.eql(conf);
     });
   });
 
@@ -80,7 +98,7 @@ describe('Print', () => {
 
       expect(getServerlessConfigFileStub.calledOnce).to.equal(true);
       expect(print.serverless.cli.consoleLog.called).to.be.equal(true);
-      expect(message).to.equal(YAML.dump(expected));
+      expect(YAML.load(message)).to.eql(expected);
     });
   });
 
@@ -115,7 +133,7 @@ describe('Print', () => {
 
       expect(getServerlessConfigFileStub.calledOnce).to.equal(true);
       expect(print.serverless.cli.consoleLog.called).to.be.equal(true);
-      expect(message).to.equal(YAML.dump(expected));
+      expect(YAML.load(message)).to.eql(expected);
     });
   });
 
@@ -154,7 +172,7 @@ describe('Print', () => {
 
       expect(getServerlessConfigFileStub.calledOnce).to.equal(true);
       expect(print.serverless.cli.consoleLog.called).to.be.equal(true);
-      expect(message).to.equal(YAML.dump(expected));
+      expect(YAML.load(message)).to.eql(expected);
     });
   });
 
@@ -186,7 +204,7 @@ describe('Print', () => {
 
       expect(getServerlessConfigFileStub.calledOnce).to.equal(true);
       expect(print.serverless.cli.consoleLog.called).to.be.equal(true);
-      expect(message).to.equal(YAML.dump(expected));
+      expect(YAML.load(message)).to.eql(expected);
     });
   });
 });