Repository: serverless/serverless (JavaScript)
Date: 2017-05-15 15:45:22+00:00
Task type: Bug Fix

Issue:
Referencing external config files using ${file()} fails and breaks Cloudformation template
# This is a Bug Report

## Description
#### What Went Wrong
PR #2842 indicates that it is possible to configure a service by referencing external resource files in `serverless.yml` as such

```
service: service-name

provider:
  name: aws
  runtime: python2.7

resources:
  - ${file(./dynamodb.yml)}
```

However, the above configuration yields an error and results in the JSON object located at  `.serverless/cloudformation-template-update-stack.json` being populated with the individual characters of the literal string `${file(./dynamodb.yml)}`

Here’s the error message that is returned:
```
Serverless: Packaging service...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading function .zip files to S3...
Serverless: Updating Stack...

  Serverless Error ---------------------------------------

     Invalid template property or properties [22, 10, 11,
     12, 13, 14, 15, 16, 17, 18, 19, 0, 1, 2, 3, 4, 5, 6,
     7, 8, 9, 20, 21]

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Forums:        forum.serverless.com
     Chat:          gitter.im/serverless/serverless

  Your Environment Information -----------------------------
     OS:                 darwin
     Node Version:       4.1.1
     Serverless Version: 1.11.0
```

and here’s what the `.serverless/cloudformation-template-update-stack.json` file looks like as a result
```
{
  "0": "$",
  "1": "{",
  "2": "f",
  "3": "i",
  "4": "l",
  "5": "e",
  "6": "(",
  "7": ".",
  "8": "/",
  "9": "d",
  "10": "y",
  "11": "n",
  "12": "a",
  "13": "m",
  "14": "o",
  "15": "d",
  "16": "b",
  "17": ".",
  "18": "y",
  "19": "m",
  "20": "l",
  "21": ")",
  "22": "}",
  "AWSTemplateFormatVersion": "2010-09-09",
  "Description": "The AWS CloudFormation template for this Serverless application",
  "Resources": {
    "ServerlessDeploymentBucket": {
      "Type": "AWS::S3::Bucket"
    }
  },
  "Outputs": {
    "ServerlessDeploymentBucketName": {
      "Value": {
        "Ref": "ServerlessDeploymentBucket"
      }
    }
  }
}
```

#### What did you expect should have happened?
The reference `${file(./dynamodb.yml)}` should have caused the Serverless Framework to load the YAML config in dynamodb.yml and use that to configure resources.

#### What was the config you used?
see description

####  What stacktrace or error message from your provider did you see?
see description

Similar or dependent issues:
* #2828

## Additional Data
* ***Serverless Framework Version you're using***: 1.11.0
* ***Operating System***: Mac OS X El Capitan 10.11.6



Issue discussion:
Hey, @pmuens 
I'd like to take a swing at fixing this bug. Can you point me in the right direction? Any idea what code is causing it?
Hey @ubaniabalogun thanks for jumping into this!

I believe it must be somewhere here in the `Variables` class: https://github.com/serverless/serverless/blob/6cca16764df1b9d43c9981642c259bd97063e93a/lib/classes/Variables.js#L173-L231

One thing to keep in mind is that @eahefnawy is currently reworking the variable system behind the scenes so it might be helpful to coordinate with him on this. Eslam also wrote the whole Variable system so he knows how this works!
I migrated today from 1.8 to 1.12.1 and I found this bug too :(.
If I copy & paste the code inside `serverless.yml` it works fine.
I just tested this on 1.13.0 and it works 👍 
I tested 1.13.0 and variables were not replaced...
serverless.yml:
```
resources:
  Resources: ${file(serverless-resources.yml)}
```

serverless-resources.yml:
```
myKmsKeyAlias:
  Type: 'AWS::KMS::Alias'
  Properties:
    AliasName: 'alias/${self:service}-helloworld'
    TargetKeyId:
      Fn::GetAtt:
        - 'myKmsKey'
        - 'Arn'
```

`$ sls package`
--> ./.serverless/cloudformation-template-update-stack.json
`        "AliasName": "alias/${self:service}-helloworld",`

@mt-sergio you are right but this is actually a different bug.  the original bug applied to the resources section, this bug has come up and causes *all* file-based variables to fail as far as I can tell - not just that they're loaded poorly as 'keys', but that the references remain unaffected

This was introduced via commit 7cadf284440447ea8774d8abb640323713485f22

Solution patch:
diff --git a/lib/Serverless.js b/lib/Serverless.js
index cf893bd8f66..2bd0c6f5b92 100644
--- a/lib/Serverless.js
+++ b/lib/Serverless.js
@@ -89,6 +89,10 @@ class Serverless {
       // (https://github.com/serverless/serverless/issues/2997)
       this.service.setFunctionNames(this.processedInput.options);
 
+      // merge custom resources after variables have been populated
+      // (https://github.com/serverless/serverless/issues/3511)
+      this.service.mergeResourceArrays();
+
       // validate the service configuration, now that variables are loaded
       this.service.validate();
 
diff --git a/lib/classes/Service.js b/lib/classes/Service.js
index eeba0a5b6bb..72026fad6f2 100644
--- a/lib/classes/Service.js
+++ b/lib/classes/Service.js
@@ -89,12 +89,6 @@ class Service {
           throw new ServerlessError(errorMessage);
         }
 
-        if (Array.isArray(serverlessFile.resources)) {
-          serverlessFile.resources = serverlessFile.resources.reduce((memo, value) =>
-            Object.assign(memo, value)
-          , {});
-        }
-
         if (_.isObject(serverlessFile.service)) {
           that.serviceObject = serverlessFile.service;
           that.service = serverlessFile.service.name;
@@ -143,6 +137,21 @@ class Service {
     });
   }
 
+  mergeResourceArrays() {
+    if (Array.isArray(this.resources)) {
+      this.resources = this.resources.reduce((memo, value) => {
+        if (value) {
+          if (typeof value === 'object') {
+            return _.merge(memo, value);
+          }
+          throw new Error(`Non-object value specified in resources array: ${value}`);
+        }
+
+        return memo;
+      }, {});
+    }
+  }
+
   validate() {
     _.forEach(this.functions, (functionObj, functionName) => {
       if (!_.isArray(functionObj.events)) {


Test patch:
diff --git a/lib/classes/Service.test.js b/lib/classes/Service.test.js
index 0493cbf7824..5c580931af0 100644
--- a/lib/classes/Service.test.js
+++ b/lib/classes/Service.test.js
@@ -170,37 +170,6 @@ describe('Service', () => {
       });
     });
 
-    it('should merge resources given as an array', () => {
-      const SUtils = new Utils();
-      const serverlessYml = {
-        service: 'new-service',
-        provider: 'aws',
-        resources: [
-          {
-            aws: {
-              resourcesProp: 'value',
-            },
-          },
-          {
-            azure: {},
-          },
-        ],
-      };
-
-      SUtils.writeFileSync(path.join(tmpDirPath, 'serverless.yml'),
-        YAML.dump(serverlessYml));
-
-      const serverless = new Serverless();
-      serverless.init();
-      serverless.config.update({ servicePath: tmpDirPath });
-      serviceInstance = new Service(serverless);
-
-      return serviceInstance.load().then(() => {
-        expect(serviceInstance.resources.aws).to.deep.equal({ resourcesProp: 'value' });
-        expect(serviceInstance.resources.azure).to.deep.equal({});
-      });
-    });
-
     it('should fail when the service name is missing', () => {
       const SUtils = new Utils();
       const serverlessYaml = {
@@ -513,6 +482,96 @@ describe('Service', () => {
     });
   });
 
+  describe('#mergeResourceArrays', () => {
+    it('should merge resources given as an array', () => {
+      const serverless = new Serverless();
+      const serviceInstance = new Service(serverless);
+
+      serviceInstance.resources = [
+        {
+          Resources: {
+            aws: {
+              resourcesProp: 'value',
+            },
+          },
+        },
+        {
+          Resources: {
+            azure: {},
+          },
+        }, {
+          foo: 'bar',
+        },
+      ];
+
+      serviceInstance.mergeResourceArrays();
+
+      expect(serviceInstance.resources).to.be.an('object');
+      expect(serviceInstance.resources.Resources).to.be.an('object');
+      expect(serviceInstance.resources.Resources.aws).to.deep.equal({ resourcesProp: 'value' });
+      expect(serviceInstance.resources.Resources.azure).to.deep.equal({});
+      expect(serviceInstance.resources.foo).to.deep.equal('bar');
+    });
+
+    it('should ignore an object', () => {
+      const serverless = new Serverless();
+      const serviceInstance = new Service(serverless);
+
+      serviceInstance.resources = {
+        Resources: 'foo',
+      };
+
+      serviceInstance.mergeResourceArrays();
+
+      expect(serviceInstance.resources).to.deep.eql({
+        Resources: 'foo',
+      });
+    });
+
+    it('should tolerate an empty string', () => {
+      const serverless = new Serverless();
+      const serviceInstance = new Service(serverless);
+
+      serviceInstance.resources = [
+        '',
+        {
+          aws: {
+            resourcesProp: 'value',
+          },
+        },
+      ];
+
+      serviceInstance.mergeResourceArrays();
+      expect(serviceInstance.resources).to.deep.eql({
+        aws: {
+          resourcesProp: 'value',
+        },
+      });
+    });
+
+    it('should throw when given a number', () => {
+      const serverless = new Serverless();
+      const serviceInstance = new Service(serverless);
+
+      serviceInstance.resources = [
+        42,
+      ];
+
+      expect(() => serviceInstance.mergeResourceArrays()).to.throw(Error);
+    });
+
+    it('should throw when given a string', () => {
+      const serverless = new Serverless();
+      const serviceInstance = new Service(serverless);
+
+      serviceInstance.resources = [
+        'string',
+      ];
+
+      expect(() => serviceInstance.mergeResourceArrays()).to.throw(Error);
+    });
+  });
+
   describe('#setFunctionNames()', () => {
     let serviceInstance;
     let tmpDirPath;