Repository: serverless/serverless (JavaScript)
Date: 2019-07-11 22:49:05+00:00
Task type: Feature

Issue:
ssm variable option to split StringList into array
# This is a Feature Proposal

## Description
SSM params can have a StringList type of comma separated values. It would be great if there was some syntax we could provide to automatically split this StringList by comma into an array. Without splitting this string value, the current Serverless variable resolution will put the entire StringList value into the CF template. For example, say if I stored a StringList param of all my `subnetIds` per account in SSM. It would be nice if I can set the vpc config value `subnetIds` to something like `${ssm:my_path_to_subnet_ids_list~true~split}`. An alternative to this explicit setting would be to always split StringList types into an array by default.



Issue discussion:
I think this might work better as a separate function, more like: `${split(${ssm:my_path_to_subnet_ids_list~true})}`... Although this gets interesting, since really the SSM string could be anything... javascript, json, yaml, etc. Some ability to process the returned string value might be valuable...
@lorengordon I agree with the function idea. I came to the same conclusion shortly after writing this issue.

Regarding your concern about SSM string containing values that could be anything, I believe that shouldn't be a problem for the StringList type since commas are already reserved as delimiters.

We would need some way of passing in a `,` delimiter into the `split(...)` function without serverless parsing the comma out first.
@lorengordon @aliatsis Hi,
I think supporting native SSM variable lists is a good addition, whilst building a custom layer ontop of it is imo not. This would create additional technical debt, and having it implemented in the SSM handling would put it into the wrong layer of indirection. Additionally, the logic used to define arrays would force people to set their SSM variables in a very specific way (the vars are most likely set and used by other implementations than Serverless too).
So, I would go for support of the native string lists.
If an array semantics would be there it should be part of the global variable system, usable for other storage systems too.

@serverless/framework-maintainers @erikerikson What are your thoughts about this?
@HyperBrain I guess I'm not sure which approach you are advocating for.

Do you want to support the StringList type directly in the SSM handler? This could be implemented by checking whether the `Type` field in the GetParameter return value is `StringList`, splitting on commas (as that appears to be the only delimiter SSM supports), and returning the list. Not hard, but useful only to SSM. (You can also bet that people will then want a `join` function to recombine the list 😄)

Or are you advocating for a more generic function, along the lines of the `file()` precedent, that could be used to split _any_ string, as returned by any of the variable handlers? Such a function would take arguments of the string to split and the delimiter to split on. Also not hard, and much more generally useful across all variable handlers. (Knowing this approach does crack open the door for more variable interpolation functions...)

I agree and disagree @HyperBrain.

String operations seem generally useful.  There's already something in there that could be expanded or augmented.  Perhaps a new `${split(<string>)}` and `${join(<string>)}` (any other obvious candidates?) would be good additions.

The disagreement...  In a sense, the particularities of SSM are entirely within scope of the SSM variable handling code.  Not allowing that could force SSM specific logic into the generic codebase.  If SSM parameters with the `StringList` type are encountered, then always splitting the returned value might be correct.  For users that didn't want the value split, the new join capability could return the data to the state they desired. Are there any other types (now, or in the future) that would need special handling?

And also agreement... encoding an automatic treatment of a subtype of SSM parameter can be interpreted as overreach.  Especially to anyone not expecting that behavior.  Extra so if we are going to provide `split` and `join` functions.  It could be considered that the lack of proper handling of commas inside parentheses in data is a bug of the override detection.  Except that this was not a handled use case previously.

I think, to make a call, that we should implement the string modification functions and make sure that the override detection handles that data within one of these functions contain a comma that should not be considered a delimiter for the purposes of override detection.

This might not be entirely thought through and my kids excited about Christmas morning so best of all to y'all.
This would be very nice, but until it's fixed, how do I parse a StringList to my config? The following didn't work:
```yaml
subnetIds: ${ssm:my_subnet_ids}
```
nor did
```yaml
subnetIds: [ ${ssm:my_subnet_ids} ] 
```
I have flexibility regarding how to write the ssm value
@vikeri did you find a solution?  Running into the same issue myself now.
@tylergohl I manually extracted each id into a separate variable and placed them in separate ssm values...
Whelp - that's what I figured.  Thanks @vikeri 
@tylergohl @vikeri  just encountered same issue, but I managed to use the following structure:
```
Condition:
  IpAddress:
    aws:SourceIp: { 'Fn::Split': [ ",", "${ssm:some_value}" ] }
```
some_value - is a StringList of coma separated IP ranges

@higef Nice!

Solution patch:
diff --git a/docs/providers/aws/guide/variables.md b/docs/providers/aws/guide/variables.md
index 9aa6cf0b181..1f814091ab8 100644
--- a/docs/providers/aws/guide/variables.md
+++ b/docs/providers/aws/guide/variables.md
@@ -294,6 +294,19 @@ custom:
 
 In this example, the serverless variable will contain the decrypted value of the SecureString.
 
+For StringList type parameters, you can optionally split the resolved variable into an array using the extended syntax, `ssm:/path/to/stringlistparam~split`.
+
+```yml
+service: new-service
+provider: aws
+functions:
+  hello:
+    name: hello
+    handler: handler.hello
+custom:
+  myArrayVar: ${ssm:/path/to/stringlistparam~split}
+```
+
 ## Reference Variables using AWS Secrets Manager
 
 Variables in [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/) can be referenced [using SSM](https://docs.aws.amazon.com/systems-manager/latest/userguide/integration-ps-secretsmanager.html). Use the `ssm:/aws/reference/secretsmanager/secret_ID_in_Secrets_Manager~true` syntax(note `~true` as secrets are always encrypted). For example:
diff --git a/lib/classes/Variables.js b/lib/classes/Variables.js
index 64c377a9a9e..685963cd1d8 100644
--- a/lib/classes/Variables.js
+++ b/lib/classes/Variables.js
@@ -54,7 +54,7 @@ class Variables {
     this.stringRefSyntax = RegExp(/(?:('|").*?\1)/g);
     this.cfRefSyntax = RegExp(/^(?:\${)?cf(\.[a-zA-Z0-9-]+)?:/g);
     this.s3RefSyntax = RegExp(/^(?:\${)?s3:(.+?)\/(.+)$/);
-    this.ssmRefSyntax = RegExp(/^(?:\${)?ssm:([a-zA-Z0-9_.\-/]+)[~]?(true|false)?/);
+    this.ssmRefSyntax = RegExp(/^(?:\${)?ssm:([a-zA-Z0-9_.\-/]+)[~]?(true|false|split)?/);
   }
 
   loadVariableSyntax() {
@@ -762,6 +762,7 @@ class Variables {
     const groups = variableString.match(this.ssmRefSyntax);
     const param = groups[1];
     const decrypt = groups[2] === 'true';
+    const split = groups[2] === 'split';
     return this.serverless
       .getProvider('aws')
       .request(
@@ -775,8 +776,10 @@ class Variables {
       ) // Use request cache
       .then(response => {
         const plainText = response.Parameter.Value;
+        const type = response.Parameter.Type;
         // Only if Secrets Manager. Parameter Store does not support JSON.
-        if (param.startsWith('/aws/reference/secretsmanager')) {
+        // We cannot parse StringList types, so don't try
+        if (type !== 'StringList' && param.startsWith('/aws/reference/secretsmanager')) {
           try {
             const json = JSON.parse(plainText);
             return BbPromise.resolve(json);
@@ -784,6 +787,14 @@ class Variables {
             // return as plain text if value is not JSON
           }
         }
+        if (split) {
+          if (type === 'StringList') {
+            return BbPromise.resolve(plainText.split(','));
+          }
+          logWarning(
+            `Cannot split SSM parameter '${param}' of type '${type}'. Must be 'StringList'.`
+          );
+        }
         return BbPromise.resolve(plainText);
       })
       .catch(err => {


Test patch:
diff --git a/lib/classes/Variables.test.js b/lib/classes/Variables.test.js
index e41352fe06d..0fc42bbd79d 100644
--- a/lib/classes/Variables.test.js
+++ b/lib/classes/Variables.test.js
@@ -2154,6 +2154,101 @@ module.exports = {
         })
         .finally(() => ssmStub.restore());
     });
+    it('should get split StringList variable from Ssm using extended syntax', () => {
+      const stringListValue = 'MockValue1,MockValue2';
+      const parsedValue = ['MockValue1', 'MockValue2'];
+      const stringListResponseMock = {
+        Parameter: {
+          Value: stringListValue,
+          Type: 'StringList',
+        },
+      };
+      const ssmStub = sinon
+        .stub(awsProvider, 'request')
+        .callsFake(() => BbPromise.resolve(stringListResponseMock));
+      return serverless.variables
+        .getValueFromSsm(`ssm:${param}~split`)
+        .should.become(parsedValue)
+        .then(() => {
+          expect(ssmStub).to.have.been.calledOnce;
+          expect(ssmStub).to.have.been.calledWithExactly(
+            'SSM',
+            'getParameter',
+            {
+              Name: param,
+              WithDecryption: false,
+            },
+            { useCache: true }
+          );
+        })
+        .finally(() => ssmStub.restore());
+    });
+    it('should get unsplit StringList variable from Ssm by default', () => {
+      const stringListValue = 'MockValue1,MockValue2';
+      const stringListResponseMock = {
+        Parameter: {
+          Value: stringListValue,
+          Type: 'StringList',
+        },
+      };
+      const ssmStub = sinon
+        .stub(awsProvider, 'request')
+        .callsFake(() => BbPromise.resolve(stringListResponseMock));
+      return serverless.variables
+        .getValueFromSsm(`ssm:${param}`)
+        .should.become(stringListValue)
+        .then(() => {
+          expect(ssmStub).to.have.been.calledOnce;
+          expect(ssmStub).to.have.been.calledWithExactly(
+            'SSM',
+            'getParameter',
+            {
+              Name: param,
+              WithDecryption: false,
+            },
+            { useCache: true }
+          );
+        })
+        .finally(() => ssmStub.restore());
+    });
+    it('should warn when attempting to split a non-StringList Ssm variable', () => {
+      const logWarningSpy = sinon.spy(slsError, 'logWarning');
+      const consoleLogStub = sinon.stub(console, 'log').returns();
+      const ProxyQuiredVariables = proxyquire('./Variables.js', { './Error': logWarningSpy });
+      const varProxy = new ProxyQuiredVariables(serverless);
+      const stringListResponseMock = {
+        Parameter: {
+          Value: value,
+          Type: 'String',
+        },
+      };
+      const ssmStub = sinon
+        .stub(awsProvider, 'request')
+        .callsFake(() => BbPromise.resolve(stringListResponseMock));
+      return varProxy
+        .getValueFromSsm(`ssm:${param}~split`)
+        .should.become(value)
+        .then(() => {
+          expect(ssmStub).to.have.been.calledOnce;
+          expect(ssmStub).to.have.been.calledWithExactly(
+            'SSM',
+            'getParameter',
+            {
+              Name: param,
+              WithDecryption: false,
+            },
+            { useCache: true }
+          );
+          expect(logWarningSpy).to.have.been.calledWithExactly(
+            `Cannot split SSM parameter '${param}' of type 'String'. Must be 'StringList'.`
+          );
+        })
+        .finally(() => {
+          ssmStub.restore();
+          logWarningSpy.restore();
+          consoleLogStub.restore();
+        });
+    });
     describe('Referencing to AWS SecretsManager', () => {
       it('should NOT parse value as json if not referencing to AWS SecretsManager', () => {
         const secretParam = '/path/to/foo-bar';