Repository: serverless/serverless (JavaScript)
Date: 2018-09-19 16:12:27+00:00
Task type: Bug Fix

Issue:
Variable file() path isn't parsed if it contains `@`
# This is a Bug Report

## Description

I'm trying to load variables from a scoped npm module in `serverless.yml`:
```yml
custom:
  variableFromFile: ${file(node_modules/@scope/module-name/variables.json):variableName}
```

But when I try to deploy the service, I get the following error:
```
TypeError: Cannot read property 'Policies' of undefined
    at functionObj.events.forEach.event (/Users/adam/.config/yarn/global/node_modules/serverless/lib/plugins/aws/package/compile/events/stream/index.js:182:18)
    at Array.forEach (<anonymous>)
    at serverless.service.getAllFunctions.forEach (/Users/adam/.config/yarn/global/node_modules/serverless/lib/plugins/aws/package/compile/events/stream/index.js:41:28)
    at Array.forEach (<anonymous>)
    at AwsCompileStreamEvents.compileStreamEvents (/Users/adam/.config/yarn/global/node_modules/serverless/lib/plugins/aws/package/compile/events/stream/index.js:16:47)
    at BbPromise.reduce (/Users/adam/.config/yarn/global/node_modules/serverless/lib/classes/PluginManager.js:366:55)
From previous event:
    at PluginManager.invoke (/Users/adam/.config/yarn/global/node_modules/serverless/lib/classes/PluginManager.js:366:22)
    at PluginManager.spawn (/Users/adam/.config/yarn/global/node_modules/serverless/lib/classes/PluginManager.js:384:17)
    at Deploy.BbPromise.bind.then.then (/Users/adam/.config/yarn/global/node_modules/serverless/lib/plugins/deploy/deploy.js:120:50)
From previous event:
    at Object.before:deploy:deploy [as hook] (/Users/adam/.config/yarn/global/node_modules/serverless/lib/plugins/deploy/deploy.js:110:10)
    at BbPromise.reduce (/Users/adam/.config/yarn/global/node_modules/serverless/lib/classes/PluginManager.js:366:55)
From previous event:
    at PluginManager.invoke (/Users/adam/.config/yarn/global/node_modules/serverless/lib/classes/PluginManager.js:366:22)
    at PluginManager.run (/Users/adam/.config/yarn/global/node_modules/serverless/lib/classes/PluginManager.js:397:17)
    at variables.populateService.then (/Users/adam/.config/yarn/global/node_modules/serverless/lib/Serverless.js:104:33)
    at runCallback (timers.js:785:20)
    at tryOnImmediate (timers.js:747:5)
    at processImmediate [as _immediateCallback] (timers.js:718:5)
From previous event:
    at Serverless.run (/Users/adam/.config/yarn/global/node_modules/serverless/lib/Serverless.js:91:74)
    at serverless.init.then (/Users/adam/.config/yarn/global/node_modules/serverless/bin/serverless:42:50)
    at <anonymous>
```

If I copy the JSON file to a path that doesn't contain the `@` symbol, it works.

## Additional Data

* ***Serverless Framework Version you're using***: 1.24.1
* ***Operating System***: OSX



Issue discussion:
Thank you for reporting @adambiggs :+1: that's a bug exactly since it has been reproduced on my end too.
I put `@test.yml` file on my test project and set up the following serverless.yml, then ran `sls print`.
However the `${file(./@test.yml):variableName}` did not be parsed properly. When removing `@`, it works fine though.

```yml
service: sls-test
provider:
  name: aws
  runtime: nodejs6.10
  stage: dev
  region: us-east-1

functions:
  hello:
    handler: handler.hello
    events:
      - http:
          path: hello
          method: get

custom:
  test: ${file(./@test.yml):variableName}
```
@adambiggs I have exactly the same problem, I'm glad you opened an issue. :+1: 
The reason for this bug is quite simple. The regex we use to parse the file variable syntax just does not allow for `@`.

`/^file\((~?[a-zA-Z0-9._\-/]+?)\)/`

So only alphanumeric characters are allowed including slashes, underscores and dashes. BTW: Using Windows path separators will also break 🤔 

We should instead use a regex that really matches file paths (including `\` and some special characters. 
We should rather go for a blacklist regex here:
`/^file\(([^?%*:|"<>]+?)\)/`

I'm not sure why the file syntax does not even allow for absolute paths. Imo we should allow these.
The regex I wrote above seems to be suitable. It allows for any valid filename.
I ran into the same problem and it took a while to figure out what was going on - hope we can get the fix in soon!  

On current project, I have an automation repo that's published as NPM package to private NPM registry under `@foo/bar` name - so I have to resort to symbolic links to work around this.


Solution patch:
diff --git a/docs/providers/aws/guide/variables.md b/docs/providers/aws/guide/variables.md
index 5cb29d33777..627dbd042b4 100644
--- a/docs/providers/aws/guide/variables.md
+++ b/docs/providers/aws/guide/variables.md
@@ -18,7 +18,7 @@ They are especially useful when providing secrets for your service to use and wh
 
 ## Syntax
 
-To use variables, you will need to reference values enclosed in `${}` brackets. 
+To use variables, you will need to reference values enclosed in `${}` brackets.
 
 ```yml
 # serverless.yml file
@@ -27,11 +27,11 @@ yamlKeyXYZ: ${variableSource} # see list of current variable sources below
 otherYamlKey: ${variableSource, defaultValue}
 ```
 
-You can define your own variable syntax (regex) if it conflicts with CloudFormation's syntax. 
+You can define your own variable syntax (regex) if it conflicts with CloudFormation's syntax.
 
 **Note:** You can only use variables in `serverless.yml` property **values**, not property keys. So you can't use variables to generate dynamic logical IDs in the custom resources section for example.
 
-## Current variable sources: 
+## Current variable sources:
 
 - [environment variables](https://serverless.com/framework/docs/providers/aws/guide/variables#referencing-environment-variables)
 - [CLI options](https://serverless.com/framework/docs/providers/aws/guide/variables#referencing-cli-options)
@@ -57,7 +57,7 @@ provider:
     MY_SECRET: ${file(./config.${self:provider.stage}.json):CREDS}
 ```
 
-If `sls deploy --stage qa` is run, the option `stage=qa` is used inside the `${file(./config.${self:provider.stage}.json):CREDS}` variable and it will resolve the `config.qa.json` file and use the `CREDS` key defined. 
+If `sls deploy --stage qa` is run, the option `stage=qa` is used inside the `${file(./config.${self:provider.stage}.json):CREDS}` variable and it will resolve the `config.qa.json` file and use the `CREDS` key defined.
 
 **How that works:**
 
@@ -427,8 +427,8 @@ service: new-service
 provider:
   name: aws
   runtime: nodejs6.10
-  variableSyntax: "\\${{([ ~:a-zA-Z0-9._\\'\",\\-\\/\\(\\)]+?)}}" # notice the double quotes for yaml to ignore the escape characters!
-#  variableSyntax: "\\${((?!AWS)[ ~:a-zA-Z0-9._'\",\\-\\/\\(\\)]+?)}" # Use this for allowing CloudFormation Pseudo-Parameters in your serverless.yml -- e.g. ${AWS::Region}. All other Serverless variables work as usual.
+  variableSyntax: "\\${{([ ~:a-zA-Z0-9._@\\'\",\\-\\/\\(\\)]+?)}}" # notice the double quotes for yaml to ignore the escape characters!
+#  variableSyntax: "\\${((?!AWS)[ ~:a-zA-Z0-9._@'\",\\-\\/\\(\\)]+?)}" # Use this for allowing CloudFormation Pseudo-Parameters in your serverless.yml -- e.g. ${AWS::Region}. All other Serverless variables work as usual.
 
 custom:
   myStage: ${{opt:stage}}
diff --git a/lib/classes/Service.js b/lib/classes/Service.js
index f4bb771d340..e37747ac208 100644
--- a/lib/classes/Service.js
+++ b/lib/classes/Service.js
@@ -19,7 +19,7 @@ class Service {
     this.provider = {
       stage: 'dev',
       region: 'us-east-1',
-      variableSyntax: '\\${([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}',
+      variableSyntax: '\\${([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}',
     };
     this.custom = {};
     this.plugins = [];
diff --git a/lib/classes/Utils.js b/lib/classes/Utils.js
index 7bdc8e62c1f..0377bc86499 100644
--- a/lib/classes/Utils.js
+++ b/lib/classes/Utils.js
@@ -252,7 +252,7 @@ class Utils {
       }
 
       let hasCustomVariableSyntaxDefined = false;
-      const defaultVariableSyntax = '\\${([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}';
+      const defaultVariableSyntax = '\\${([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}';
 
       // check if the variableSyntax in the provider section is defined
       if (provider && provider.variableSyntax
diff --git a/lib/classes/Variables.js b/lib/classes/Variables.js
index 6411f043f70..2b60f047c84 100644
--- a/lib/classes/Variables.js
+++ b/lib/classes/Variables.js
@@ -35,7 +35,7 @@ const PromiseTracker = require('./PromiseTracker');
  *   adornments that the framework's components make to the originally loaded service.  As a result,
  *   it is important to reset this object for each use.
  * 3. Note that despite some AWS code herein that this class is used in all plugins.  Obviously
- *   users avoid the AWS-specific variable types when targetting other clouds.
+ *   users avoid the AWS-specific variable types when targeting other clouds.
  */
 class Variables {
   constructor(serverless) {
@@ -46,7 +46,7 @@ class Variables {
     this.deep = [];
     this.deepRefSyntax = RegExp(/(\${)?deep:\d+(\.[^}]+)*()}?/);
     this.overwriteSyntax = RegExp(/\s*(?:,\s*)+/g);
-    this.fileRefSyntax = RegExp(/^file\((~?[a-zA-Z0-9._\-/]+?)\)/g);
+    this.fileRefSyntax = RegExp(/^file\(([^?%*:|"<>]+?)\)/g);
     this.envRefSyntax = RegExp(/^env:/g);
     this.optRefSyntax = RegExp(/^opt:/g);
     this.selfRefSyntax = RegExp(/^self:/g);
diff --git a/lib/plugins/print/print.js b/lib/plugins/print/print.js
index 9c146bdbc18..b6760437f94 100644
--- a/lib/plugins/print/print.js
+++ b/lib/plugins/print/print.js
@@ -54,7 +54,7 @@ class Print {
     service.provider = _.merge({
       stage: 'dev',
       region: 'us-east-1',
-      variableSyntax: '\\${([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}',
+      variableSyntax: '\\${([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}',
     }, service.provider);
   }
   strip(svc) {


Test patch:
diff --git a/lib/classes/Service.test.js b/lib/classes/Service.test.js
index 446208a2f5b..9c124447b2d 100644
--- a/lib/classes/Service.test.js
+++ b/lib/classes/Service.test.js
@@ -31,7 +31,7 @@ describe('Service', () => {
       expect(serviceInstance.provider).to.deep.equal({
         stage: 'dev',
         region: 'us-east-1',
-        variableSyntax: '\\${([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}',
+        variableSyntax: '\\${([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}',
       });
       expect(serviceInstance.custom).to.deep.equal({});
       expect(serviceInstance.plugins).to.deep.equal([]);
@@ -131,7 +131,7 @@ describe('Service', () => {
           name: 'aws',
           stage: 'dev',
           region: 'us-east-1',
-          variableSyntax: '\\${{([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}}',
+          variableSyntax: '\\${{([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}}',
         },
         plugins: ['testPlugin'],
         functions: {
@@ -163,7 +163,7 @@ describe('Service', () => {
         expect(serviceInstance.service).to.be.equal('new-service');
         expect(serviceInstance.provider.name).to.deep.equal('aws');
         expect(serviceInstance.provider.variableSyntax).to.equal(
-          '\\${{([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}}'
+          '\\${{([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}}'
         );
         expect(serviceInstance.plugins).to.deep.equal(['testPlugin']);
         expect(serviceInstance.resources.aws).to.deep.equal({ resourcesProp: 'value' });
@@ -186,7 +186,7 @@ describe('Service', () => {
           name: 'aws',
           stage: 'dev',
           region: 'us-east-1',
-          variableSyntax: '\\${{([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}}',
+          variableSyntax: '\\${{([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}}',
         },
         plugins: ['testPlugin'],
         functions: {
@@ -217,7 +217,7 @@ describe('Service', () => {
         expect(serviceInstance.service).to.be.equal('new-service');
         expect(serviceInstance.provider.name).to.deep.equal('aws');
         expect(serviceInstance.provider.variableSyntax).to.equal(
-          '\\${{([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}}'
+          '\\${{([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}}'
         );
         expect(serviceInstance.plugins).to.deep.equal(['testPlugin']);
         expect(serviceInstance.resources.aws).to.deep.equal({ resourcesProp: 'value' });
@@ -240,7 +240,7 @@ describe('Service', () => {
           name: 'aws',
           stage: 'dev',
           region: 'us-east-1',
-          variableSyntax: '\\${{([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}}',
+          variableSyntax: '\\${{([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}}',
         },
         plugins: ['testPlugin'],
         functions: {
@@ -272,7 +272,7 @@ describe('Service', () => {
         expect(serviceInstance.service).to.be.equal('new-service');
         expect(serviceInstance.provider.name).to.deep.equal('aws');
         expect(serviceInstance.provider.variableSyntax).to.equal(
-          '\\${{([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}}'
+          '\\${{([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}}'
         );
         expect(serviceInstance.plugins).to.deep.equal(['testPlugin']);
         expect(serviceInstance.resources.aws).to.deep.equal({ resourcesProp: 'value' });
@@ -295,7 +295,7 @@ describe('Service', () => {
           name: 'aws',
           stage: 'dev',
           region: 'us-east-1',
-          variableSyntax: '\\${{([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}}',
+          variableSyntax: '\\${{([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}}',
         },
         plugins: ['testPlugin'],
         functions: {
@@ -327,7 +327,7 @@ describe('Service', () => {
         expect(serviceInstance.service).to.be.equal('new-service');
         expect(serviceInstance.provider.name).to.deep.equal('aws');
         expect(serviceInstance.provider.variableSyntax).to.equal(
-          '\\${{([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}}'
+          '\\${{([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}}'
         );
         expect(serviceInstance.plugins).to.deep.equal(['testPlugin']);
         expect(serviceInstance.resources.aws).to.deep.equal({ resourcesProp: 'value' });
@@ -363,7 +363,7 @@ describe('Service', () => {
           name: 'aws',
           stage: 'dev',
           region: 'us-east-1',
-          variableSyntax: '\\${{([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}}',
+          variableSyntax: '\\${{([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}}',
         },
         plugins: ['testPlugin'],
         functions: {
@@ -792,7 +792,7 @@ describe('Service', () => {
           name: 'aws',
           stage: 'dev',
           region: 'us-east-1',
-          variableSyntax: '\\${{([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}}',
+          variableSyntax: '\\${{([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}}',
         },
         plugins: ['testPlugin'],
         functions: {
diff --git a/lib/classes/Variables.test.js b/lib/classes/Variables.test.js
index 19c68a8e324..92422255627 100644
--- a/lib/classes/Variables.test.js
+++ b/lib/classes/Variables.test.js
@@ -48,7 +48,7 @@ describe('Variables', () => {
   describe('#loadVariableSyntax()', () => {
     it('should set variableSyntax', () => {
       // eslint-disable-next-line no-template-curly-in-string
-      serverless.service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}}';
+      serverless.service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}}';
       serverless.variables.loadVariableSyntax();
       expect(serverless.variables.variableSyntax).to.be.a('RegExp');
     });
@@ -295,7 +295,7 @@ describe('Variables', () => {
       beforeEach(() => {
         service = makeDefault();
         // eslint-disable-next-line no-template-curly-in-string
-        service.provider.variableSyntax = '\\${([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}'; // default
+        service.provider.variableSyntax = '\\${([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}'; // default
         serverless.variables.service = service;
         serverless.variables.loadVariableSyntax();
         delete service.provider.variableSyntax;
@@ -635,7 +635,7 @@ describe('Variables', () => {
           .should.become(expected);
       });
       it('should handle deep variables regardless of custom variableSyntax', () => {
-        service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._\\\'",\\-\\/\\(\\)]+?)}}';
+        service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._@\\\'",\\-\\/\\(\\)]+?)}}';
         serverless.variables.loadVariableSyntax();
         delete service.provider.variableSyntax;
         service.custom = {
@@ -652,7 +652,7 @@ describe('Variables', () => {
           .should.become(expected);
       });
       it('should handle deep variables regardless of recursion into custom variableSyntax', () => {
-        service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._\\\'",\\-\\/\\(\\)]+?)}}';
+        service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._@\\\'",\\-\\/\\(\\)]+?)}}';
         serverless.variables.loadVariableSyntax();
         delete service.provider.variableSyntax;
         service.custom = {
@@ -673,7 +673,7 @@ describe('Variables', () => {
           .should.become(expected);
       });
       it('should handle deep variables in complex recursions of custom variableSyntax', () => {
-        service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._\\\'",\\-\\/\\(\\)]+?)}}';
+        service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._@\\\'",\\-\\/\\(\\)]+?)}}';
         serverless.variables.loadVariableSyntax();
         delete service.provider.variableSyntax;
         service.custom = {
@@ -704,7 +704,7 @@ describe('Variables', () => {
           fse.removeSync(tmpDirPath);
         });
         const makeTempFile = (fileName, fileContent) => {
-          fse.writeFileSync(path.join(tmpDirPath, fileName), fileContent);
+          fse.outputFileSync(path.join(tmpDirPath, fileName), fileContent);
         };
         const asyncFileName = 'async.load.js';
         const asyncContent = `'use strict';
@@ -792,6 +792,23 @@ module.exports = {
             return serverless.variables.populateObject(service.custom)
               .should.become(expected);
           });
+        it('should populate variables from filesnames including \'@\',  e.g scoped npm packages',
+          () => {
+            const fileName = `./node_modules/@scoped-org/${asyncFileName}`;
+            makeTempFile(
+                fileName,
+                asyncContent
+            );
+            service.custom = {
+              val0: `\${file(${fileName}):str}`,
+            };
+            const expected = {
+              val0: 'my-async-value-1',
+            };
+            return serverless.variables
+                .populateObject(service.custom)
+                .should.become(expected);
+          });
         const selfFileName = 'self.yml';
         const selfContent = `foo: baz
 bar: \${self:custom.self.foo}
@@ -1025,7 +1042,7 @@ module.exports = {
 
   describe('#overwrite()', () => {
     beforeEach(() => {
-      serverless.service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}}';
+      serverless.service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}}';
       serverless.variables.loadVariableSyntax();
       delete serverless.service.provider.variableSyntax;
     });
@@ -1243,7 +1260,7 @@ module.exports = {
 
   describe('#getValueFromSelf()', () => {
     beforeEach(() => {
-      serverless.service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}}';
+      serverless.service.provider.variableSyntax = '\\${{([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}}';
       serverless.variables.loadVariableSyntax();
       delete serverless.service.provider.variableSyntax;
     });
diff --git a/lib/plugins/print/print.test.js b/lib/plugins/print/print.test.js
index ccd2cd25f12..31c1b2f2616 100644
--- a/lib/plugins/print/print.test.js
+++ b/lib/plugins/print/print.test.js
@@ -32,7 +32,7 @@ describe('Print', () => {
   });
 
   afterEach(() => {
-    serverless.service.provider.variableSyntax = '\\${([ ~:a-zA-Z0-9._\'",\\-\\/\\(\\)]+?)}';
+    serverless.service.provider.variableSyntax = '\\${([ ~:a-zA-Z0-9._@\'",\\-\\/\\(\\)]+?)}';
   });
 
   it('should print standard config', () => {
@@ -256,10 +256,10 @@ describe('Print', () => {
       provider: {
         name: 'aws',
         stage: '${{opt:stage}}',
-        variableSyntax: "\\${{([ ~:a-zA-Z0-9._\\'\",\\-\\/\\(\\)]+?)}}",
+        variableSyntax: "\\${{([ ~:a-zA-Z0-9._@\\'\",\\-\\/\\(\\)]+?)}}",
       },
     };
-    serverless.service.provider.variableSyntax = "\\${{([ ~:a-zA-Z0-9._\\'\",\\-\\/\\(\\)]+?)}}";
+    serverless.service.provider.variableSyntax = "\\${{([ ~:a-zA-Z0-9._@\\'\",\\-\\/\\(\\)]+?)}}";
     getServerlessConfigFileStub.resolves(conf);
 
     serverless.processedInput = {
@@ -272,7 +272,7 @@ describe('Print', () => {
       provider: {
         name: 'aws',
         stage: 'dev',
-        variableSyntax: "\\${{([ ~:a-zA-Z0-9._\\'\",\\-\\/\\(\\)]+?)}}",
+        variableSyntax: "\\${{([ ~:a-zA-Z0-9._@\\'\",\\-\\/\\(\\)]+?)}}",
       },
     };