Repository: serverless/serverless (JavaScript)
Date: 2018-05-18 07:11:01+00:00
Task type: Bug Fix

Issue:
Custom deep variables (still) appear not to be correctly resolved
This is a Bug Report
====================

Description
-----------

A follow up to  #4946
(and https://github.com/amplify-education/serverless-domain-manager/issues/124)

For bug reports:

-   What went wrong?\
    My deep custom serverless variables aren't resolved properly.

-   What did you expect should have happened?\
    The variables should be resolved correctly.

-   What was the config you used?\
    Using this config:

Config relevant base part:

```
plugins:
  - serverless-domain-manager

provider:
    stage: ${opt:stage, self:custom.defaultStage}

custom:
  defaultStage: dev

  customDomain:
    domainName: ${self:custom.apiFQDN}
```

and 2 different custom configs:


1)
```
custom
  SLD: mydomain.com
  apiHost: api
  hostPostfix:
    dev: -dev
    staging: -staging
    test: -test
  apiFQDN: ${self:custom.apiHost}${self:custom.hostPostfix.${self:provider.stage}, ''}.${self:custom.SLD}

```

2)
```
custom:
  SLD: mydomain.com
  hostPostfix:
    dev: -dev
    staging: -staging
    test: -test
  apiHost: api${self:custom.hostPostfix.${self:provider.stage}, ''}
  apiFQDN: ${self:custom.apiHost}.${self:custom.SLD}

  
```


The .serverless/serverless-state.json looks good in both variants:

```
"customDomain": {
  "domainName": "api-staging.mydomain.com",
  ...
}
```
But running:

```
sls info --stage staging

```
The CLI output on 1 is
```
Error --------------------------------------------------
 
Error: Domain manager summary logging failed.
Error: Error: 'self:custom.apiHost.self:custom.SLD' could not be found in API Gateway.
NotFoundException: Invalid domain name identifier specified
```

The CLI output on 2 is
```
Error --------------------------------------------------
 
Error: Domain manager summary logging failed.
Error: Error: 'api.mydomain.com' could not be found in API Gateway.
NotFoundException: Invalid domain name identifier specified
```
-   What stacktrace or error message from your provider did you see?

```
NotFoundException: Invalid domain name identifier specified
    at getDomain.then.catch (/path/project/node_modules/serverless-domain-manager/index.js:225:13)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:228:7)
From previous event:
    at PluginManager.invoke (/path/project/node_modules/serverless/lib/classes/PluginManager.js:372:22)
    at PluginManager.run (/path/project/node_modules/serverless/lib/classes/PluginManager.js:403:17)
    at variables.populateService.then (/path/project/node_modules/serverless/lib/Serverless.js:102:33)
    at runCallback (timers.js:789:20)
    at tryOnImmediate (timers.js:751:5)
    at processImmediate [as _immediateCallback] (timers.js:722:5)
From previous event:
    at Serverless.run (/path/project/node_modules/serverless/lib/Serverless.js:89:74)
    at serverless.init.then (/path/project/node_modules/serverless/bin/serverless:42:50)
    at <anonymous>

```

Additional Data
---------------

-   ***Serverless Framework Version you're using***: 1.27.2
-   ***Operating System***: macos
-   ***Stack Trace***: above
-   ***Provider Error messages***: ?

Custom deep variables (still) appear not to be correctly resolved
This is a Bug Report
====================

Description
-----------

A follow up to  #4946
(and https://github.com/amplify-education/serverless-domain-manager/issues/124)

For bug reports:

-   What went wrong?\
    My deep custom serverless variables aren't resolved properly.

-   What did you expect should have happened?\
    The variables should be resolved correctly.

-   What was the config you used?\
    Using this config:

Config relevant base part:

```
plugins:
  - serverless-domain-manager

provider:
    stage: ${opt:stage, self:custom.defaultStage}

custom:
  defaultStage: dev

  customDomain:
    domainName: ${self:custom.apiFQDN}
```

and 2 different custom configs:


1)
```
custom
  SLD: mydomain.com
  apiHost: api
  hostPostfix:
    dev: -dev
    staging: -staging
    test: -test
  apiFQDN: ${self:custom.apiHost}${self:custom.hostPostfix.${self:provider.stage}, ''}.${self:custom.SLD}

```

2)
```
custom:
  SLD: mydomain.com
  hostPostfix:
    dev: -dev
    staging: -staging
    test: -test
  apiHost: api${self:custom.hostPostfix.${self:provider.stage}, ''}
  apiFQDN: ${self:custom.apiHost}.${self:custom.SLD}

  
```


The .serverless/serverless-state.json looks good in both variants:

```
"customDomain": {
  "domainName": "api-staging.mydomain.com",
  ...
}
```
But running:

```
sls info --stage staging

```
The CLI output on 1 is
```
Error --------------------------------------------------
 
Error: Domain manager summary logging failed.
Error: Error: 'self:custom.apiHost.self:custom.SLD' could not be found in API Gateway.
NotFoundException: Invalid domain name identifier specified
```

The CLI output on 2 is
```
Error --------------------------------------------------
 
Error: Domain manager summary logging failed.
Error: Error: 'api.mydomain.com' could not be found in API Gateway.
NotFoundException: Invalid domain name identifier specified
```
-   What stacktrace or error message from your provider did you see?

```
NotFoundException: Invalid domain name identifier specified
    at getDomain.then.catch (/path/project/node_modules/serverless-domain-manager/index.js:225:13)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:228:7)
From previous event:
    at PluginManager.invoke (/path/project/node_modules/serverless/lib/classes/PluginManager.js:372:22)
    at PluginManager.run (/path/project/node_modules/serverless/lib/classes/PluginManager.js:403:17)
    at variables.populateService.then (/path/project/node_modules/serverless/lib/Serverless.js:102:33)
    at runCallback (timers.js:789:20)
    at tryOnImmediate (timers.js:751:5)
    at processImmediate [as _immediateCallback] (timers.js:722:5)
From previous event:
    at Serverless.run (/path/project/node_modules/serverless/lib/Serverless.js:89:74)
    at serverless.init.then (/path/project/node_modules/serverless/bin/serverless:42:50)
    at <anonymous>

```

Additional Data
---------------

-   ***Serverless Framework Version you're using***: 1.27.2
-   ***Operating System***: macos
-   ***Stack Trace***: above
-   ***Provider Error messages***: ?

Custom deep variables (still) appear not to be correctly resolved
This is a Bug Report
====================

Description
-----------

A follow up to  #4946
(and https://github.com/amplify-education/serverless-domain-manager/issues/124)

For bug reports:

-   What went wrong?\
    My deep custom serverless variables aren't resolved properly.

-   What did you expect should have happened?\
    The variables should be resolved correctly.

-   What was the config you used?\
    Using this config:

Config relevant base part:

```
plugins:
  - serverless-domain-manager

provider:
    stage: ${opt:stage, self:custom.defaultStage}

custom:
  defaultStage: dev

  customDomain:
    domainName: ${self:custom.apiFQDN}
```

and 2 different custom configs:


1)
```
custom
  SLD: mydomain.com
  apiHost: api
  hostPostfix:
    dev: -dev
    staging: -staging
    test: -test
  apiFQDN: ${self:custom.apiHost}${self:custom.hostPostfix.${self:provider.stage}, ''}.${self:custom.SLD}

```

2)
```
custom:
  SLD: mydomain.com
  hostPostfix:
    dev: -dev
    staging: -staging
    test: -test
  apiHost: api${self:custom.hostPostfix.${self:provider.stage}, ''}
  apiFQDN: ${self:custom.apiHost}.${self:custom.SLD}

  
```


The .serverless/serverless-state.json looks good in both variants:

```
"customDomain": {
  "domainName": "api-staging.mydomain.com",
  ...
}
```
But running:

```
sls info --stage staging

```
The CLI output on 1 is
```
Error --------------------------------------------------
 
Error: Domain manager summary logging failed.
Error: Error: 'self:custom.apiHost.self:custom.SLD' could not be found in API Gateway.
NotFoundException: Invalid domain name identifier specified
```

The CLI output on 2 is
```
Error --------------------------------------------------
 
Error: Domain manager summary logging failed.
Error: Error: 'api.mydomain.com' could not be found in API Gateway.
NotFoundException: Invalid domain name identifier specified
```
-   What stacktrace or error message from your provider did you see?

```
NotFoundException: Invalid domain name identifier specified
    at getDomain.then.catch (/path/project/node_modules/serverless-domain-manager/index.js:225:13)
    at <anonymous>
    at process._tickDomainCallback (internal/process/next_tick.js:228:7)
From previous event:
    at PluginManager.invoke (/path/project/node_modules/serverless/lib/classes/PluginManager.js:372:22)
    at PluginManager.run (/path/project/node_modules/serverless/lib/classes/PluginManager.js:403:17)
    at variables.populateService.then (/path/project/node_modules/serverless/lib/Serverless.js:102:33)
    at runCallback (timers.js:789:20)
    at tryOnImmediate (timers.js:751:5)
    at processImmediate [as _immediateCallback] (timers.js:722:5)
From previous event:
    at Serverless.run (/path/project/node_modules/serverless/lib/Serverless.js:89:74)
    at serverless.init.then (/path/project/node_modules/serverless/bin/serverless:42:50)
    at <anonymous>

```

Additional Data
---------------

-   ***Serverless Framework Version you're using***: 1.27.2
-   ***Operating System***: macos
-   ***Stack Trace***: above
-   ***Provider Error messages***: ?



Issue discussion:
Just tried another plugin

```
plugins:
  - serverless-scriptable-plugin

provider:
  ...
  environment:
    MYTEST: ${self:custom.apiFQDN}

custom:
  scriptHooks:
    'package:createDeploymentArtifacts': myscript.js 

```

and

```
// myscript.js
// gets passed serverless object

console.log(serverless.service.provider.environment.MYTEST)
```

which also behaves identically, showing the same incorrect value for MYTEST.
I am having the exact same issue, evaluating a deep variable in that format (trying to solve basically the exact same problem actually).  I am using the same version as the OP, but am on Windows.

My serverless.yml file can be found here, https://github.com/PinsterTeam/Frontend/blob/serverless-bug/serverless.yml
@bebbi @stretch1414 This is only related to the `info` command?

/cc @erikerikson 
Mine happens on package/deploy.  The variable is not evaluated properly when being referenced elsewhere in the yml, but if I look in the json generated by package, the variable is the value I expect.
Thank you very much for your reports.  It appears that the fix for harvesting variable syntax from the given example variables (that are being replaced by deep variables) is improperly doing the harvesting in these less straight-forward cases.  Investigating in depth...
Just tried another plugin

```
plugins:
  - serverless-scriptable-plugin

provider:
  ...
  environment:
    MYTEST: ${self:custom.apiFQDN}

custom:
  scriptHooks:
    'package:createDeploymentArtifacts': myscript.js 

```

and

```
// myscript.js
// gets passed serverless object

console.log(serverless.service.provider.environment.MYTEST)
```

which also behaves identically, showing the same incorrect value for MYTEST.
I am having the exact same issue, evaluating a deep variable in that format (trying to solve basically the exact same problem actually).  I am using the same version as the OP, but am on Windows.

My serverless.yml file can be found here, https://github.com/PinsterTeam/Frontend/blob/serverless-bug/serverless.yml
@bebbi @stretch1414 This is only related to the `info` command?

/cc @erikerikson 
Mine happens on package/deploy.  The variable is not evaluated properly when being referenced elsewhere in the yml, but if I look in the json generated by package, the variable is the value I expect.
Thank you very much for your reports.  It appears that the fix for harvesting variable syntax from the given example variables (that are being replaced by deep variables) is improperly doing the harvesting in these less straight-forward cases.  Investigating in depth...
Just tried another plugin

```
plugins:
  - serverless-scriptable-plugin

provider:
  ...
  environment:
    MYTEST: ${self:custom.apiFQDN}

custom:
  scriptHooks:
    'package:createDeploymentArtifacts': myscript.js 

```

and

```
// myscript.js
// gets passed serverless object

console.log(serverless.service.provider.environment.MYTEST)
```

which also behaves identically, showing the same incorrect value for MYTEST.
I am having the exact same issue, evaluating a deep variable in that format (trying to solve basically the exact same problem actually).  I am using the same version as the OP, but am on Windows.

My serverless.yml file can be found here, https://github.com/PinsterTeam/Frontend/blob/serverless-bug/serverless.yml
@bebbi @stretch1414 This is only related to the `info` command?

/cc @erikerikson 
Mine happens on package/deploy.  The variable is not evaluated properly when being referenced elsewhere in the yml, but if I look in the json generated by package, the variable is the value I expect.
Thank you very much for your reports.  It appears that the fix for harvesting variable syntax from the given example variables (that are being replaced by deep variables) is improperly doing the harvesting in these less straight-forward cases.  Investigating in depth...

Solution patch:
diff --git a/lib/classes/Variables.js b/lib/classes/Variables.js
index 28f62386ae7..7cebde7aa7c 100644
--- a/lib/classes/Variables.js
+++ b/lib/classes/Variables.js
@@ -729,12 +729,8 @@ class Variables {
     if (index < 0) {
       index = this.deep.push(variable) - 1;
     }
-    let variableContainer = variable;
-    let variableString = this.cleanVariable(variableContainer);
-    while (variableString.match(this.variableSyntax)) {
-      variableContainer = variableString;
-      variableString = this.cleanVariable(variableContainer);
-    }
+    const variableContainer = variable.match(this.variableSyntax)[0];
+    const variableString = this.cleanVariable(variableContainer);
     return variableContainer
       .replace(/\s/g, '')
       .replace(variableString, `deep:${index}`);


Test patch:
diff --git a/lib/classes/Variables.test.js b/lib/classes/Variables.test.js
index c03bceeef19..1c8ec69eb24 100644
--- a/lib/classes/Variables.test.js
+++ b/lib/classes/Variables.test.js
@@ -624,6 +624,27 @@ describe('Variables', () => {
         return serverless.variables.populateObject(service.custom)
           .should.become(expected);
       });
+      it('should handle deep variables in complex recursions of custom variableSyntax', () => {
+        service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._\\\'",\\-\\/\\(\\)]+?)}}';
+        serverless.variables.loadVariableSyntax();
+        delete service.provider.variableSyntax;
+        service.custom = {
+          i0: '0',
+          s0: 'DEV',
+          s1: '${{self:custom.s0}}! ${{self:custom.s0}}',
+          s2: 'I am a ${{self:custom.s0}}! A ${{self:custom.s${{self:custom.i0}}}}!',
+          s3: '${{self:custom.s0}}!, I am a ${{self:custom.s1}}!, ${{self:custom.s2}}',
+        };
+        const expected = {
+          i0: '0',
+          s0: 'DEV',
+          s1: 'DEV! DEV',
+          s2: 'I am a DEV! A DEV!',
+          s3: 'DEV!, I am a DEV! DEV!, I am a DEV! A DEV!',
+        };
+        return serverless.variables.populateObject(service.custom)
+          .should.become(expected);
+      });
       describe('file reading cases', () => {
         let tmpDirPath;
         beforeEach(() => {