Repository: serverless/serverless (JavaScript)
Date: 2018-06-06 19:10:36+00:00
Task type: Bug Fix

Issue:
json-cycle is causing problems with $ref in CF template
# This is a Bug Report

## Description

If a serverless plugin reads some configuration from the .yml file and writes it in to the template multiple times (for example, for each function, create a resource) _and_ it has a property that is a complex type (such as `Fn::Join`), `json-cycle` sees these as two copies of the 'exact same' (`===`) object and replaces all but the first with `$ref`. This is invalid CF syntax.

The workaround is to _.cloneDeep everything you're inserting in to the compiledCloudformationTemplate. This isn't ideal, I don't think CF can *ever* support cyclic JSON anyway, so writing it out using `json-cycle` seems like it only breaks things at best.

And as far as I can tell, there isn't even a cycle. `json-cycle` is actually doing more than it should.

I guess serverless could clone the template deep in its entirety but I don't think it's going to matter or help.

## Additional Data

This is a self-contained reproducible example: https://github.com/dougmoscrop/serverless-example-json-ref-issue

* ***Serverless Framework Version you're using***:
latest; bug does NOT happen with 1.24.1

* ***Operating System***: OSX
* ***Stack Trace***: N/A
* ***Provider Error messages***: N/A



Issue discussion:
This issue is preventing deployments without hard coding a string literal value for properties we're using `join`, `sub`, or `ref`, etc for.

I've been manually deploying outside our pipelines, which obviously is an absolute last resort workaround.
Just to reiterate, hopefully in brief:

- If you have a cycle, `json-cycle` will *always* output invalid CloudFormation
- If you do not have a cycle `json-cycle` will *sometimes* output invalid CloudFormation that would otherwise stringify just fine

I see no reason to use `json-cycle` in any methods dealing with writing file output.
<img width="360" alt="screen shot 2018-06-05 at 11 50 48 am" src="https://user-images.githubusercontent.com/1812955/40996467-b9552cac-68b6-11e8-870e-55638adbdf6a.png">

@dougmoscrop, I took a glance through the uses of `json-cycle` in serverless and it's in the core utils for writing files.

Are you proposing it's safe to remove `json-cycle` step altogether, without adverse side-effects?

Seems like the solutions are:
- `_.cloneDeep` everything being inserted into `compiledCloudFormationTemplate` (not ideal)
- Use [`json-cycle#decycle`](https://www.npmjs.com/package/json-cycle#decycle) method to parse back the `$ref`s to full qualified values (which seems redundant, compressing data just to almost immediately decompress?)
- Remove `json-cycle` altogether

Couldn't figure out off-hand what purpose it serves...
Could it have been added as a premature optimization?

I'd like to take a stab at fixing this issue and opening a PR.
@rkdavidson `json-cycle` is IMO used to serialize the serverless state (the `this.serverless.service` object). This object can have self-references at any level if you use the `self:service` variable in your configuration, which is quite common. I am sure that everything serializing and deserializing will break if you remove it without a proper replacement (in fact at all locations you posted above).
Thanks for chiming in @HyperBrain, that makes sense — actually just read the comments in code about that, so I should have thought about that.

In that case, could a `decycle` call before outputting final compiled cloud formation template solve?
Decycyle is the problem.

It's not replacing "duplicate references that are part of a cycle" it's replacing *all* duplicate references with $ref. 

There's no scenario I can think of where the utility provided by json-cycle will be of use when dealing with the cf templates. Other serializations.. maybe, sure... But the cf template output is at best only sometimes broken by json-cycle.


Solution patch:
diff --git a/lib/classes/Utils.js b/lib/classes/Utils.js
index 8010d88b304..c0b7bb4545c 100644
--- a/lib/classes/Utils.js
+++ b/lib/classes/Utils.js
@@ -39,15 +39,15 @@ class Utils {
     return fse.mkdirsSync(path.dirname(filePath));
   }
 
-  writeFileSync(filePath, contents) {
-    return writeFileSync(filePath, contents);
+  writeFileSync(filePath, contents, cycles) {
+    return writeFileSync(filePath, contents, cycles);
   }
 
-  writeFile(filePath, contents) {
+  writeFile(filePath, contents, cycles) {
     const that = this;
     return new BbPromise((resolve, reject) => {
       try {
-        that.writeFileSync(filePath, contents);
+        that.writeFileSync(filePath, contents, cycles);
       } catch (e) {
         reject(e);
       }
diff --git a/lib/plugins/aws/package/lib/saveServiceState.js b/lib/plugins/aws/package/lib/saveServiceState.js
index 46756bd6271..7d6afe2fbb6 100644
--- a/lib/plugins/aws/package/lib/saveServiceState.js
+++ b/lib/plugins/aws/package/lib/saveServiceState.js
@@ -36,7 +36,7 @@ module.exports = {
       },
     };
 
-    this.serverless.utils.writeFileSync(serviceStateFilePath, state);
+    this.serverless.utils.writeFileSync(serviceStateFilePath, state, true);
 
     return BbPromise.resolve();
   },
diff --git a/lib/utils/fs/writeFile.js b/lib/utils/fs/writeFile.js
index 625bdbc285c..abeef07e0e7 100644
--- a/lib/utils/fs/writeFile.js
+++ b/lib/utils/fs/writeFile.js
@@ -5,13 +5,17 @@ const path = require('path');
 const jc = require('json-cycle');
 const YAML = require('js-yaml');
 
-function writeFile(filePath, conts) {
+function writeFile(filePath, conts, cycles) {
   let contents = conts || '';
 
   return fse.mkdirsAsync(path.dirname(filePath))
     .then(() => {
       if (filePath.indexOf('.json') !== -1 && typeof contents !== 'string') {
-        contents = jc.stringify(contents, null, 2);
+        if (cycles) {
+          contents = jc.stringify(contents, null, 2);
+        } else {
+          contents = JSON.stringify(contents, null, 2);
+        }
       }
 
       const yamlFileExists = (filePath.indexOf('.yaml') !== -1);
diff --git a/lib/utils/fs/writeFileSync.js b/lib/utils/fs/writeFileSync.js
index ab5fff093ee..6e58263c41b 100644
--- a/lib/utils/fs/writeFileSync.js
+++ b/lib/utils/fs/writeFileSync.js
@@ -5,13 +5,17 @@ const path = require('path');
 const jc = require('json-cycle');
 const YAML = require('js-yaml');
 
-function writeFileSync(filePath, conts) {
+function writeFileSync(filePath, conts, cycles) {
   let contents = conts || '';
 
   fse.mkdirsSync(path.dirname(filePath));
 
   if (filePath.indexOf('.json') !== -1 && typeof contents !== 'string') {
-    contents = jc.stringify(contents, null, 2);
+    if (cycles) {
+      contents = jc.stringify(contents, null, 2);
+    } else {
+      contents = JSON.stringify(contents, null, 2);
+    }
   }
 
   const yamlFileExists = (filePath.indexOf('.yaml') !== -1);


Test patch:
diff --git a/lib/plugins/aws/package/lib/saveServiceState.test.js b/lib/plugins/aws/package/lib/saveServiceState.test.js
index db9b02bd1f0..064fb861c0f 100644
--- a/lib/plugins/aws/package/lib/saveServiceState.test.js
+++ b/lib/plugins/aws/package/lib/saveServiceState.test.js
@@ -63,7 +63,7 @@ describe('#saveServiceState()', () => {
       };
 
       expect(getServiceStateFileNameStub.calledOnce).to.equal(true);
-      expect(writeFileSyncStub.calledWithExactly(filePath, expectedStateFileContent))
+      expect(writeFileSyncStub.calledWithExactly(filePath, expectedStateFileContent, true))
         .to.equal(true);
     });
   });
@@ -97,7 +97,7 @@ describe('#saveServiceState()', () => {
       };
 
       expect(getServiceStateFileNameStub.calledOnce).to.equal(true);
-      expect(writeFileSyncStub.calledWithExactly(filePath, expectedStateFileContent))
+      expect(writeFileSyncStub.calledWithExactly(filePath, expectedStateFileContent, true))
         .to.equal(true);
     });
   });
diff --git a/lib/utils/fs/writeFile.test.js b/lib/utils/fs/writeFile.test.js
index 972669f40c4..966c928d43c 100644
--- a/lib/utils/fs/writeFile.test.js
+++ b/lib/utils/fs/writeFile.test.js
@@ -49,7 +49,7 @@ describe('#writeFile()', function () {
     bar.foo = bar;
     const expected = '{\n  "foo": {\n    "$ref": "$"\n  }\n}';
 
-    return writeFile(tmpFilePath, bar)
+    return writeFile(tmpFilePath, bar, true)
       .then(() =>
         expect(fse.readFileAsync(tmpFilePath, 'utf8')).to.eventually.equal(expected)
       );
diff --git a/lib/utils/fs/writeFileSync.test.js b/lib/utils/fs/writeFileSync.test.js
index 64cc604b931..87ec760a57e 100644
--- a/lib/utils/fs/writeFileSync.test.js
+++ b/lib/utils/fs/writeFileSync.test.js
@@ -53,7 +53,7 @@ describe('#writeFileSync()', () => {
     bar.foo = bar;
     const expected = '{\n  "foo": {\n    "$ref": "$"\n  }\n}';
 
-    writeFileSync(tmpFilePath, bar);
+    writeFileSync(tmpFilePath, bar, true);
 
     return fse.readFileAsync(tmpFilePath, 'utf8').then((contents) => {
       expect(contents).to.equal(expected);