Repository: alexcasalboni/aws-lambda-power-tuning
Pull request #107 (2020-07-23 13:47:46)

Implement payloadS3 input parameter to fetch payloads from Amazon S3
Implement #106 

TODO:
- [x] Basic implementation
- [x] Basic tests
- [x] Implement dynamic IAM permissions at deploy-time
- [x] Add more unit tests (e.g. weighted payloads)
- [x] Integration tests
- [x] Update documentation
- [x] Verify and document the impact on the tuning process (difference in cost / performance?)


Patch:
diff --git a/README-DEPLOY.md b/README-DEPLOY.md
index a7e3b18..02f6736 100644
--- a/README-DEPLOY.md
+++ b/README-DEPLOY.md
@@ -16,6 +16,8 @@ If you don't want to deal with any Infrastructure as Code tool, you can use one
 1.  The [Lambda Power Tuner UI](#option5) 
 
 
+Read more about the [deployment parameters here](README-INPUT-OUTPUT.md#state-machine-configuration-at-deployment-time)
+
 ## Option 1: AWS Serverless Application Repository<a name="option1"></a>
 
 You can find this app in the [Serverless Application Repository](https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:451282441545:applications~aws-lambda-power-tuning) and deploy it with just a few clicks in the AWS Management Console.
diff --git a/README-INPUT-OUTPUT.md b/README-INPUT-OUTPUT.md
index 12621ad..ca2b350 100644
--- a/README-INPUT-OUTPUT.md
+++ b/README-INPUT-OUTPUT.md
@@ -8,9 +8,10 @@ Each execution of the state machine will require an input and will provide the c
 The state machine accepts the following intput parameters:
 
 * **lambdaARN** (required, string): unique identifier of the Lambda function you want to optimize
-* **powerValues** (optional, string or list of integers): the list of power values to be tested; if not provided, the default values configured at deploy-time are used (by default: 128MB, 256MB, 512MB, 1024MB, 1536MB, and 3008MB); you can provide any power values between 128MB and 3,008MB in 64 MB increments; if you provide the string `"ALL"` instead of a list, all possible power configurations will be tested
+* **powerValues** (optional, string or list of integers): the list of power values to be tested; if not provided, the default values configured at deploy-time are used (by default: 128MB, 256MB, 512MB, 1024MB, 1536MB, and 3008MB); you can provide any power values between 128MB and 10,240MB
 * **num** (required, integer): the # of invocations for each power configuration (minimum 5, recommended: between 10 and 100)
 * **payload** (string, object, or list): the static payload that will be used for every invocation (object or string); when using a list, a weighted payload is expected in the shape of `[{"payload": {...}, "weight": X }, {"payload": {...}, "weight": Y }, {"payload": {...}, "weight": Z }]`, where the weights `X`, `Y`, and `Z` are treated as relative weights (not perentages); more details below in the [Weighted Payloads section](#user-content-weighted-payloads)
+* **payloadS3** (string): a reference to Amazon S3 for large payloads (>256KB), formatted as `s3://bucket/key`; it requires read-only IAM permissions, see `payloadS3Bucket` and `payloadS3Key` below and find more details in the [S3 payloads section](#user-content-s3-payloads)
 * **parallelInvocation** (false by default): if true, all the invocations will be executed in parallel (note: depending on the value of `num`, you may experience throttling when setting `parallelInvocation` to true)
 * **strategy** (string): it can be `"cost"` or `"speed"` or `"balanced"` (the default value is `"cost"`); if you use `"cost"` the state machine will suggest the cheapest option (disregarding its performance), while if you use `"speed"` the state machine will suggest the fastest option (disregarding its cost). When using `"balanced"` the state machine will choose a compromise between `"cost"` and `"speed"` according to the parameter `"balancedWeight"`
 * **balancedWeight** (number between 0.0 and 1.0, by default is 0.5): parameter that express the trade-off between cost and time, 0.0 is equivalent to `"speed"` strategy, 1.0 is equivalent to `"cost"` strategy
@@ -21,7 +22,7 @@ The state machine accepts the following intput parameters:
 * **postProcessorARN** (string): it must be the ARN of a Lambda function; if provided, the function will be invoked after every invocation of `lambdaARN`; more details below in the [Pre/Post-processing functions section](#user-content-prepost-processing-functions)
 
 
-## State machine input (at deployment time)
+## State machine configuration (at deployment time)
 
 The CloudFormation template accepts the following parameters:
 
@@ -30,6 +31,8 @@ The CloudFormation template accepts the following parameters:
 * **totalExecutionTimeout** (number in seconds, default=`300`): the timeout in seconds applied to all functions of the state machine
 * **lambdaResource** (string, default=`*`): the `Resource` used in IAM policies; it's `*` by default but you could restrict it to a prefix or a specific function ARN
 * **permissionsBoundary** (string): the ARN of a permissions boundary (policy), applied to all functions of the state machine
+* **payloadS3Bucket** (string): the S3 bucket name used for large payloads (>256KB); if provided, it's added to a custom managed IAM policy that grants read-only permission to the S3 bucket; more details below in the [S3 payloads section](#user-content-s3-payloads)
+* **payloadS3Key** (string, default=`*`): they S3 object key used for large payloads (>256KB); the default value grants access to all S3 objects in the bucket specified with `payloadS3Bucket`; more details below in the [S3 payloads section](#user-content-s3-payloads)
 
 Please note that the total execution time should stay below 300 seconds (5 min), which is the default timeout. You can easily estimate the total execution timout based on the average duration of your functions. For example, if your function's average execution time is 5 seconds and you haven't enabled `parallelInvocation`, you should set `totalExecutionTimeout` to at least `num * 5`: 50 seconds if `num=10`, 500 seconds if `num=100`, and so on. If you have enabled `parallelInvocation`, usually you don't need to tune the value of `totalExecutionTimeout` unless your average execution time is above 5 min.
 
@@ -48,22 +51,21 @@ If you want to run the state machine as part of your continuous integration pipe
 }
 ```
 
-Of course, you can use different alias names such as `dev`, `test`, `production`, etc.
-
-If you don't configure any alias name, the state machine will only update the `$LATEST` alias.
+You can use different alias names such as `dev`, `test`, `production`, etc. If you don't configure any alias name, the state machine will only update the `$LATEST` alias.
 
 
 ### Weighted Payloads
 
-Weighted payloads can be used in scenarios where the payload structure and the corresponding performance/speed can vary a lot in production and you'd like to include multiple payloads in the tuning process.
+Weighted payloads can be used in scenarios where the payload structure and the corresponding performance/speed could vary a lot in production and you'd like to include multiple payloads in the tuning process.
 
 You may want to use weighted payloads also in case of functions with side effects that would be hard or impossible to test with the very same payload (for example, a function that deletes records from a database).
 
-You can use weighted payloads as follows in the execution input:
+You configure weighted payloads as follows:
 
 ```json
 {
     ...
+    "num": 50,
     "payload": [
         { "payload": {...}, "weight": 5 },
         { "payload": {...}, "weight": 15 },
@@ -72,7 +74,7 @@ You can use weighted payloads as follows in the execution input:
 }
 ```
 
-In the example above the weights `5`, `15` and `30` are used as relative weights. They will correspond to `10%` (5 out of 50), `30%` (15 out of 50), and `60%` (30 out of 50) respectively - meaning that the corresponding payload will be used 10%, 30% and 60% of the time.
+In the example above, the weights `5`, `15`, and `30` are used as relative weights. They will correspond to `10%` (5 out of 50), `30%` (15 out of 50), and `60%` (30 out of 50) respectively - meaning that the corresponding payload will be used 10%, 30% and 60% of the time.
 
 For example, if `num=100` the first payload will be used 10 times, the second 30 times, and the third 60 times.
 
@@ -107,6 +109,25 @@ Please also keep in mind the following:
 * Pre/post-processors don't have to be in the same region of the main function
 * Pre/post-processors don't alter the statistics related to cost and performance
 
+### S3 payloads
+
+In case of very large payloads above 256KB, you can provide an S3 object reference (`s3://bucket/key`) instead of an inline payload.
+
+Your state machine input will look like this:
+
+```json
+{
+    "lambdaARN": "your-lambda-function-arn",
+    "powerValues": [128, 256, 512, 1024],
+    "num": 50,
+    "payloadS3": "s3://your-bucket/your-object.json"
+}
+```
+
+Please note that the state machine will require IAM access to your S3 bucket, so you might need to redeploy the Lambda Power Tuning application and configure the `payloadS3Bucket` parameter at deployment time. This will automatically generate a custom IAM managed policy to grant read-only access to that bucket. If you want to narrow down the read-only policy to a specific object or pattern, use the `payloadS3Key` parameter (which is `*` by default).
+
+S3 payloads work fine with weighted payloads too.
+
 
 ## State Machine Output
 
diff --git a/README-SAR.md b/README-SAR.md
index eff7ab9..63a264f 100644
--- a/README-SAR.md
+++ b/README-SAR.md
@@ -35,14 +35,15 @@ As soon as you click "**Start Execution**" again, you'll be able to visualize th
 Once the execution has completed, you will find the execution results in the "**Output**" tab of the "**Execution Details**" section. The output will contain the optimal power configuration and its corresponding average cost per execution.
 
 
-## State Machine Input
+## State machine input (at execution time)
 
-The AWS Step Functions state machine accepts the following parameters:
+The state machine accepts the following intput parameters:
 
 * **lambdaARN** (required, string): unique identifier of the Lambda function you want to optimize
-* **powerValues** (optional, string or list of integers): the list of power values to be tested; if not provided, the default values configured at deploy-time are used (by default: 128MB, 256MB, 512MB, 1024MB, 1536MB, and 3008MB); you can provide any power values between 128MB and 3,008MB in 64 MB increments; if you provide the string `"ALL"` instead of a list, all possible power configurations will be tested
+* **powerValues** (optional, string or list of integers): the list of power values to be tested; if not provided, the default values configured at deploy-time are used (by default: 128MB, 256MB, 512MB, 1024MB, 1536MB, and 3008MB); you can provide any power values between 128MB and 10,240MB
 * **num** (required, integer): the # of invocations for each power configuration (minimum 5, recommended: between 10 and 100)
 * **payload** (string, object, or list): the static payload that will be used for every invocation (object or string); when using a list, a weighted payload is expected in the shape of `[{"payload": {...}, "weight": X }, {"payload": {...}, "weight": Y }, {"payload": {...}, "weight": Z }]`, where the weights `X`, `Y`, and `Z` are treated as relative weights (not perentages); more details below in the Weighted Payloads section
+* **payloadS3** (string): a reference to Amazon S3 for large payloads (>256KB), formatted as `s3://bucket/key`; it requires read-only IAM permissions, see `payloadS3Bucket` and `payloadS3Key` below and find more details in the S3 payloads section
 * **parallelInvocation** (false by default): if true, all the invocations will be executed in parallel (note: depending on the value of `num`, you may experience throttling when setting `parallelInvocation` to true)
 * **strategy** (string): it can be `"cost"` or `"speed"` or `"balanced"` (the default value is `"cost"`); if you use `"cost"` the state machine will suggest the cheapest option (disregarding its performance), while if you use `"speed"` the state machine will suggest the fastest option (disregarding its cost). When using `"balanced"` the state machine will choose a compromise between `"cost"` and `"speed"` according to the parameter `"balancedWeight"`
 * **balancedWeight** (number between 0.0 and 1.0, by default is 0.5): parameter that express the trade-off between cost and time, 0.0 is equivalent to `"speed"` strategy, 1.0 is equivalent to `"cost"` strategy
@@ -53,9 +54,19 @@ The AWS Step Functions state machine accepts the following parameters:
 * **postProcessorARN** (string): it must be the ARN of a Lambda function; if provided, the function will be invoked after every invocation of `lambdaARN`; more details below in the Pre/Post-processing functions section
 
 
-Additionally, you can specify a list of power values at deploy-time in the `PowerValues` CloudFormation parameter. These power values will be used as the default in case no `powerValues` input parameter is provided.
+## State machine configuration (at deployment time)
 
-Please note that the total execution time should stay below 300 seconds (5 min), which is the default timeout. You can customize this timeout at deploy time with the `totalExecutionTimeout` CloudFormation parameter. You can easily estimate the total execution timout based on the average duration of your functions. For example, if your function's average execution time is 5 seconds and you haven't enabled `parallelInvocation`, you should set `totalExecutionTimeout` to at least `num * 5`: 50 seconds if `num=10`, 500 seconds if `num=100`, and so on. If you have enabled `parallelInvocation`, usually you don't need to tune the value of `totalExecutionTimeout` unless your average execution time is above 5 min.
+The CloudFormation template accepts the following parameters:
+
+* **PowerValues** (list of numbers): these power values will be used as the default in case no `powerValues` input parameter is provided at execution time
+* **visualizationURL** (string): the base URL for the visualization tool, by default it's `lambda-power-tuning.show` but you could use your own custom tool
+* **totalExecutionTimeout** (number in seconds, default=`300`): the timeout in seconds applied to all functions of the state machine
+* **lambdaResource** (string, default=`*`): the `Resource` used in IAM policies; it's `*` by default but you could restrict it to a prefix or a specific function ARN
+* **permissionsBoundary** (string): the ARN of a permissions boundary (policy), applied to all functions of the state machine
+* **payloadS3Bucket** (string): the S3 bucket name used for large payloads (>256KB); if provided, it's added to a custom managed IAM policy that grants read-only permission to the S3 bucket; more details below in the [S3 payloads section](#user-content-s3-payloads)
+* **payloadS3Key** (string, default=`*`): they S3 object key used for large payloads (>256KB); the default value grants access to all S3 objects in the bucket specified with `payloadS3Bucket`; more details below in the [S3 payloads section](#user-content-s3-payloads)
+
+Please note that the total execution time should stay below 300 seconds (5 min), which is the default timeout. You can easily estimate the total execution timout based on the average duration of your functions. For example, if your function's average execution time is 5 seconds and you haven't enabled `parallelInvocation`, you should set `totalExecutionTimeout` to at least `num * 5`: 50 seconds if `num=10`, 500 seconds if `num=100`, and so on. If you have enabled `parallelInvocation`, usually you don't need to tune the value of `totalExecutionTimeout` unless your average execution time is above 5 min.
 
 ### Usage in CI/CD pipelines
 
@@ -72,21 +83,20 @@ If you want to run the state machine as part of your continuous integration pipe
 }
 ```
 
-Of course, you can use different alias names such as `dev`, `test`, `production`, etc.
-
-If you don't configure any alias name, the state machine will only update the `$LATEST` alias.
+You can use different alias names such as `dev`, `test`, `production`, etc. If you don't configure any alias name, the state machine will only update the `$LATEST` alias.
 
 ### Weighted Payloads
 
-Weighted payloads can be used in scenarios where the payload structure and the corresponding performance/speed can vary a lot in production and you'd like to include multiple payloads in the tuning process.
+Weighted payloads can be used in scenarios where the payload structure and the corresponding performance/speed could vary a lot in production and you'd like to include multiple payloads in the tuning process.
 
 You may want to use weighted payloads also in case of functions with side effects that would be hard or impossible to test with the very same payload (for example, a function that deletes records from a database).
 
-You can use weighted payloads as follows in the execution input:
+You configure weighted payloads as follows:
 
 ```json
 {
     ...
+    "num": 50,
     "payload": [
         { "payload": {...}, "weight": 5 },
         { "payload": {...}, "weight": 15 },
@@ -95,7 +105,7 @@ You can use weighted payloads as follows in the execution input:
 }
 ```
 
-In the example above the weights `5`, `15` and `30` are used as relative weights. They will correspond to `10%` (5 out of 50), `30%` (15 out of 50), and `60%` (30 out of 50) respectively - meaning that the corresponding payload will be used 10%, 30% and 60% of the time.
+In the example above, the weights `5`, `15`, and `30` are used as relative weights. They will correspond to `10%` (5 out of 50), `30%` (15 out of 50), and `60%` (30 out of 50) respectively - meaning that the corresponding payload will be used 10%, 30% and 60% of the time.
 
 For example, if `num=100` the first payload will be used 10 times, the second 30 times, and the third 60 times.
 
@@ -130,6 +140,26 @@ Please also keep in mind the following:
 * Pre/post-processors don't have to be in the same region of the main function
 * Pre/post-processors don't alter the statistics related to cost and performance
 
+### S3 payloads
+
+In case of very large payloads above 256KB, you can provide an S3 object reference (`s3://bucket/key`) instead of an inline payload.
+
+Your state machine input will look like this:
+
+```json
+{
+    "lambdaARN": "your-lambda-function-arn",
+    "powerValues": [128, 256, 512, 1024],
+    "num": 50,
+    "payloadS3": "s3://your-bucket/your-object.json"
+}
+```
+
+Please note that the state machine will require IAM access to your S3 bucket, so you might need to redeploy the Lambda Power Tuning application and configure the `payloadS3Bucket` parameter at deployment time. This will automatically generate a custom IAM managed policy to grant read-only access to that bucket. If you want to narrow down the read-only policy to a specific object or pattern, use the `payloadS3Key` parameter (which is `*` by default).
+
+S3 payloads work fine with weighted payloads too.
+
+
 ## State Machine Output
 
 The state machine will return the following output:
diff --git a/lambda/executor.js b/lambda/executor.js
index 9d6e409..221df4d 100644
--- a/lambda/executor.js
+++ b/lambda/executor.js
@@ -25,7 +25,7 @@ module.exports.handler = async(event, context) => {
         dryRun,
         preProcessorARN,
         postProcessorARN,
-    } = extractDataFromInput(event);
+    } = await extractDataFromInput(event);
 
     validateInput(lambdaARN, value, num); // may throw
 
@@ -69,14 +69,24 @@ const validateInput = (lambdaARN, value, num) => {
     }
 };
 
-const extractDataFromInput = (event) => {
+const extractPayloadValue = async(input) => {
+    if (input.payloadS3) {
+        return await utils.fetchPayloadFromS3(input.payloadS3); // might throw if access denied or 404
+    } else if (input.payload) {
+        return input.payload;
+    }
+    return null;
+};
+
+const extractDataFromInput = async(event) => {
     const input = event.input; // original state machine input
+    const payload = await extractPayloadValue(input);
     return {
         value: parseInt(event.value, 10),
         lambdaARN: input.lambdaARN,
         num: parseInt(input.num, 10),
         enableParallel: !!input.parallelInvocation,
-        payload: input.payload,
+        payload: payload,
         dryRun: input.dryRun === true,
         preProcessorARN: input.preProcessorARN,
         postProcessorARN: input.postProcessorARN,
diff --git a/lambda/utils.js b/lambda/utils.js
index c8e05d5..a1cf943 100644
--- a/lambda/utils.js
+++ b/lambda/utils.js
@@ -1,6 +1,8 @@
 'use strict';
 
 const AWS = require('aws-sdk');
+const url = require('url');
+
 
 // local reference to this module
 const utils = module.exports;
@@ -278,6 +280,63 @@ module.exports.invokeLambda = (lambdaARN, alias, payload) => {
     return lambda.invoke(params).promise();
 };
 
+/**
+ * Fetch the body of an S3 object, given an S3 path such as s3://BUCKET/KEY
+ */
+module.exports.fetchPayloadFromS3 = async(s3Path) => {
+    console.log('Fetch payload from S3', s3Path);
+
+    if (typeof s3Path !== 'string' || s3Path.indexOf('s3://') === -1) {
+        throw new Error('Invalid S3 path, not a string in the format s3://BUCKET/KEY');
+    }
+
+    const URI = url.parse(s3Path);
+    URI.pathname = decodeURIComponent(URI.pathname || '');
+
+    const bucket = URI.hostname;
+    const key = URI.pathname.slice(1);
+
+    if (!bucket || !key) {
+        throw new Error(`Invalid S3 path: "${s3Path}" (bucket: ${bucket}, key: ${key})`);
+    }
+
+    const data = await utils._fetchS3Object(bucket, key);
+
+    try {
+        // try to parse into JSON object
+        return JSON.parse(data);
+    } catch (_) {
+        // otherwise return as is
+        return data;
+    }
+
+
+};
+
+module.exports._fetchS3Object = async(bucket, key) => {
+    const s3 = new AWS.S3();
+    try {
+        const response = await s3.getObject({
+            Bucket: bucket,
+            Key: key,
+        }).promise();
+        return response.Body.toString('utf-8');
+    } catch (err) {
+        if (err.statusCode === 403) {
+            throw new Error(
+                `Permission denied when trying to read s3://${bucket}/${key}. ` +
+                'You might need to re-deploy the app with the correct payloadS3Bucket parameter.',
+            );
+        } else if (err.statusCode === 404) {
+            throw new Error(
+                `The object s3://${bucket}/${key} does not exist. ` +
+                'Make sure you are trying to access an existing object in the correct bucket.',
+            );
+        } else {
+            throw new Error(`Unknown error when trying to read s3://${bucket}/${key}. ${err.message}`);
+        }
+    }
+};
 
 /**
  * Generate a list of `num` payloads (repeated or weighted)
@@ -403,7 +462,7 @@ module.exports.computeAverageDuration = (durations) => {
 
     // compute trimmed mean (discard 20% of low/high values)
     const averageDuration = durations
-        .sort(function (a, b) {  return a - b;  }) // sort numerically
+        .sort(function(a, b) { return a - b; }) // sort numerically
         .slice(toBeDiscarded, -toBeDiscarded) // discard first/last values
         .reduce((a, b) => a + b, 0) // sum all together
         / newN
diff --git a/package-lock.json b/package-lock.json
index 09a7e8a..fedef3a 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -1843,6 +1843,14 @@
         "extsprintf": "1.3.0",
         "json-schema": "0.4.0",
         "verror": "1.10.0"
+      },
+      "dependencies": {
+        "json-schema": {
+          "version": "0.4.0",
+          "resolved": "https://registry.npmjs.org/json-schema/-/json-schema-0.4.0.tgz",
+          "integrity": "sha512-es94M3nTIfsEPisRafak+HDLfHXnKBhV3vU5eqPcS3flIWqcxJWgXHXiey3YrpaNsanY5ei1VoYEbOzijuq9BA==",
+          "dev": true
+        }
       }
     },
     "just-extend": {
diff --git a/template.yml b/template.yml
index c736e39..70c11ab 100644
--- a/template.yml
+++ b/template.yml
@@ -19,8 +19,6 @@ Parameters:
   PowerValues: 
     Type: List<Number>
     Default: 128,256,512,1024,1536,3008
-    # Default: 128,192,256,320,384,448,512,576,640,704,768,832,896,960,1024,1088,1152,1216,1280,1344,1408,1472,1536,3008
-    # AllowedValues: ['128','192','256','320','384','448','512','576','640','704','768','832','896','960','1024','1088','1152','1216','1280','1344','1408','1472','1536','3008']
     Description: Default RAM values, used only if not provided as execution input (comma-separated).
   visualizationURL:
     Type: String
@@ -40,9 +38,18 @@ Parameters:
     Type: String
     Default: ''
     Description: The ARN of a permissions boundary to use for all functions' execution role.
+  payloadS3Bucket:
+    Type: String
+    Default: ''
+    Description: S3 bucket name used for large payloads (optional), included in IAM policy if not an empty string
+  payloadS3Key:
+    Type: String
+    Default: '*'
+    Description: S3 object key used for large payloads (optional), included in IAM policy if bucket is not an empty string (all objects are readable by default)
 
 Conditions:
   UsePermissionsBoundary: !Not [!Equals [!Ref permissionsBoundary, '']]
+  S3BucketProvided: !Not [!Equals [!Ref payloadS3Bucket, '']]
 
 Globals:
   Function:
@@ -81,8 +88,8 @@ Resources:
       Layers:
         - !Ref SDKlayer
       Policies:
-        - AWSLambdaExecute # Managed Policy
-        - Version: '2012-10-17' # Policy Document
+        - AWSLambdaBasicExecutionRole # Only logs
+        - Version: '2012-10-17' # allow Lambda actions
           Statement:
             - Effect: Allow
               Action:
@@ -102,14 +109,34 @@ Resources:
       Layers:
         - !Ref SDKlayer
       Policies:
-        - AWSLambdaExecute # Managed Policy
-        - Version: '2012-10-17' # Policy Document
+        - AWSLambdaBasicExecutionRole # only logs
+        - Version: '2012-10-17' # allow Lambda actions
           Statement:
             - Effect: Allow
               Action:
                 - lambda:InvokeFunction
                 - lambda:GetFunctionConfiguration
               Resource: !Ref lambdaResource
+        - !If
+          - S3BucketProvided # if S3 bucket is provided
+          - !Ref payloadS3Policy # allow read-only access to S3
+          - !Ref AWS::NoValue
+
+  payloadS3Policy:
+    Condition: S3BucketProvided
+    Type: AWS::IAM::ManagedPolicy
+    Properties:
+      Description: Lambda Power Tunning - S3 Read Only for large payloads
+      PolicyDocument:
+        Version: '2012-10-17'
+        Statement:
+          - Effect: Allow
+            Action:
+              - s3:ListBucket # this is needed only to detect 404 errors correctly
+              - s3:GetObject
+            Resource:
+              - !Sub arn:${AWS::Partition}:s3:::${payloadS3Bucket}
+              - !Sub arn:${AWS::Partition}:s3:::${payloadS3Bucket}/${payloadS3Key} # payloadS3Key is * by default
 
   cleaner:
     Type: AWS::Serverless::Function
@@ -119,8 +146,8 @@ Resources:
       Layers:
         - !Ref SDKlayer
       Policies:
-        - AWSLambdaExecute # Managed Policy
-        - Version: '2012-10-17' # Policy Document
+        - AWSLambdaBasicExecutionRole # only logs
+        - Version: '2012-10-17' # allow Lambda actions
           Statement:
             - Effect: Allow
               Action:
@@ -136,7 +163,7 @@ Resources:
       Handler: analyzer.handler
       Timeout: 10
       Policies:
-        - AWSLambdaExecute # Managed Policy
+        - AWSLambdaBasicExecutionRole # only logs
 
   optimizer:
     Type: AWS::Serverless::Function
@@ -146,8 +173,8 @@ Resources:
       Layers:
         - !Ref SDKlayer
       Policies:
-        - AWSLambdaExecute # Managed Policy
-        - Version: '2012-10-17' # Policy Document
+        - AWSLambdaBasicExecutionRole # only logs
+        - Version: '2012-10-17' # allow Lambda actions
           Statement:
             - Effect: Allow
               Action:


Test patch:
diff --git a/test/unit/test-lambda.js b/test/unit/test-lambda.js
index 8fdc5c3..ddae7d4 100644
--- a/test/unit/test-lambda.js
+++ b/test/unit/test-lambda.js
@@ -71,7 +71,8 @@ var getLambdaAliasStub,
     invokeLambdaProcessorStub,
     deleteLambdaAliasStub,
     waitForFunctionUpdateStub,
-    getLambdaArchitectureStub;
+    getLambdaArchitectureStub,
+    fetchPayloadFromS3Stub;
 
 /** unit tests below **/
 
@@ -140,45 +141,34 @@ describe('Lambda Functions', async() => {
 
         const handler = require('../../lambda/initializer').handler;
 
-        it('should explode if invoked without a lambdaARN', async() => {
-            const invalidEvents = [
-                null,
-                {},
-                { lambdaARN: null },
-                { lambdaARN: '' },
-                { lambdaARN: false },
-                { lambdaARN: 0 },
-            ];
-            invalidEvents.forEach(event => {
-                expect(async() => {
-                    await invokeForFailure(handler, event);
-                }).to.not.throwError();
+        let invalidEvents = [
+            null,
+            {},
+            { lambdaARN: null },
+            { lambdaARN: '' },
+            { lambdaARN: false },
+            { lambdaARN: 0 },
+        ];
+
+        invalidEvents.forEach(async(event) => {
+            it('should explode if invoked without a lambdaARN - ' + JSON.stringify(event), async() => {
+                await invokeForFailure(handler, event);
             });
-
-            expect(async() => {
-                await invokeForSuccess(handler, { lambdaARN: 'arnOK', num: 5 });
-            }).to.not.throwError();
         });
 
-        it('should explode if invoked with a low num', async() => {
-            const invalidEvents = [
-                { num: -1, lambdaARN: 'arnOK' },
-                { num: 0, lambdaARN: 'arnOK' },
-                { num: 1, lambdaARN: 'arnOK' },
-                { num: 2, lambdaARN: 'arnOK' },
-                { num: 3, lambdaARN: 'arnOK' },
-                { num: 4, lambdaARN: 'arnOK' },
-            ];
-            invalidEvents.forEach(event => {
-                expect(async() => {
-                    await invokeForFailure(handler, event);
-                }).to.not.throwError();
-            });
-
-            expect(async() => {
-                await invokeForSuccess(handler, { lambdaARN: 'arnOK', num: 5 });
-            }).to.not.throwError();
+        invalidEvents = [
+            { num: -1, lambdaARN: 'arnOK' },
+            { num: 0, lambdaARN: 'arnOK' },
+            { num: 1, lambdaARN: 'arnOK' },
+            { num: 2, lambdaARN: 'arnOK' },
+            { num: 3, lambdaARN: 'arnOK' },
+            { num: 4, lambdaARN: 'arnOK' },
+        ];
 
+        invalidEvents.forEach(async(event) => {
+            it('should explode if invoked with a low num - ' + JSON.stringify(event), async() => {
+                await invokeForFailure(handler, event);
+            });
         });
 
         it('should invoke the given cb without powerValues as input', async() => {
@@ -268,26 +258,23 @@ describe('Lambda Functions', async() => {
 
         const handler = require('../../lambda/cleaner').handler;
 
-        it('should explode if invoked without a lambdaARN', async() => {
-            const invalidEvents = [
-                null,
-                {},
-                { lambdaARN: null },
-                { lambdaARN: '' },
-                { lambdaARN: false },
-                { lambdaARN: 0 },
-            ];
-            invalidEvents.forEach(async(event) => {
-                expect(async() => {
-                    await invokeForFailure(handler, event);
-                }).to.not.throwError();
+        let invalidEvents = [
+            null,
+            {},
+            { lambdaARN: null },
+            { lambdaARN: '' },
+            { lambdaARN: false },
+            { lambdaARN: 0 },
+        ];
+
+        invalidEvents.forEach(async(event) => {
+            it('should explode if invoked without a lambdaARN - ' + JSON.stringify(event), async() => {
+                await invokeForFailure(handler, event);
             });
         });
 
         it('should explode if invoked without powerValues', async() => {
-            expect(async() => {
-                await invokeForFailure(handler, {lambdaARN: 'arnOK'});
-            }).to.not.throwError();
+            await invokeForFailure(handler, {lambdaARN: 'arnOK'});
         });
 
         beforeEach('mock utilities', () => {
@@ -356,12 +343,14 @@ describe('Lambda Functions', async() => {
         var invokeLambdaCounter,
             invokeLambdaPayloads,
             invokeProcessorCounter,
+            fetchPayloadFromS3Counter,
             getLambdaArchitectureCounter;
 
         beforeEach('mock utilities', () => {
             invokeLambdaCounter = 0;
             invokeLambdaPayloads = [];
             invokeProcessorCounter = 0;
+            fetchPayloadFromS3Counter = 0;
             getLambdaArchitectureCounter = 0;
 
             invokeLambdaStub && invokeLambdaStub.restore();
@@ -394,36 +383,40 @@ describe('Lambda Functions', async() => {
                     return Math.floor(Math.random() * 2) === 0 ? 'x86_64' : 'arm64';
                 });
 
+            fetchPayloadFromS3Stub && fetchPayloadFromS3Stub.restore();
+            fetchPayloadFromS3Stub = sandBox.stub(utils, 'fetchPayloadFromS3')
+                .callsFake(async(_arn, _alias, payload) => {
+                    fetchPayloadFromS3Counter++;
+                    return {ValueFromS3: 'OK'};
+                });
         });
 
-        it('should explode if invoked with invalid input', async() => {
-            const invalidEvents = [
-                null,
-                {},
-                { lambdaARN: null },
-                { lambdaARN: '' },
-                { lambdaARN: false },
-                { lambdaARN: 0 },
-                { lambdaARN: 'arnOK' },
-                { input: {lambdaARN: null} },
-                { input: {lambdaARN: ''} },
-                { input: {lambdaARN: false} },
-                { input: {lambdaARN: 0} },
-                { input: {lambdaARN: 'arnOK'} },
-                { input: {lambdaARN: 'arnOK'}, value: null },
-                { input: {lambdaARN: 'arnOK'}, value: 0 },
-                { input: {lambdaARN: 'arnOK'}, value: 'invalid' },
-                { input: {lambdaARN: 'arnOK'}, value: 128 }, // 128 is ok
-                { input: {lambdaARN: 'arnOK'}, value: '128' }, // '128' is ok
-                { input: {lambdaARN: 'arnOK', num: null}, value: 128 },
-                { input: {lambdaARN: 'arnOK', num: 0}, value: 128 },
-                { input: {lambdaARN: 'arnOK', num: 'invalid'}, value: 128 },
-            ];
-
-            invalidEvents.forEach(event => {
-                expect(async() => {
-                    await invokeForFailure(handler, event);
-                }).to.not.throwError();
+        let invalidEvents = [
+            null,
+            {},
+            { lambdaARN: null },
+            { lambdaARN: '' },
+            { lambdaARN: false },
+            { lambdaARN: 0 },
+            { lambdaARN: 'arnOK' },
+            { input: {lambdaARN: null} },
+            { input: {lambdaARN: ''} },
+            { input: {lambdaARN: false} },
+            { input: {lambdaARN: 0} },
+            { input: {lambdaARN: 'arnOK'} },
+            { input: {lambdaARN: 'arnOK'}, value: null },
+            { input: {lambdaARN: 'arnOK'}, value: 0 },
+            { input: {lambdaARN: 'arnOK'}, value: 'invalid' },
+            { input: {lambdaARN: 'arnOK'}, value: 128 }, // 128 is ok
+            { input: {lambdaARN: 'arnOK'}, value: '128' }, // '128' is ok
+            { input: {lambdaARN: 'arnOK', num: null}, value: 128 },
+            { input: {lambdaARN: 'arnOK', num: 0}, value: 128 },
+            { input: {lambdaARN: 'arnOK', num: 'invalid'}, value: 128 },
+        ];
+
+        invalidEvents.forEach(async(event) => {
+            it('should explode if invoked with invalid input - ' + JSON.stringify(event), async() => {
+                await invokeForFailure(handler, event);
             });
         });
 
@@ -1015,24 +1008,90 @@ describe('Lambda Functions', async() => {
 
         });
 
+        it('should fetch payload from S3 if payloadS3 is given', async() => {
+
+            await invokeForSuccess(handler, {
+                value: '128',
+                input: {
+                    lambdaARN: 'arnOK',
+                    num: 50,
+                    payloadS3: 's3://my-bucket/my-key.json',
+                },
+            });
+            expect(fetchPayloadFromS3Counter).to.be(1);
+            for (let payload of invokeLambdaPayloads){
+                expect(payload).to.be('{"ValueFromS3":"OK"}');
+            }
+        });
+
+        it('should fetch payload from S3 if both payload and payloadS3 are given', async() => {
+
+            await invokeForSuccess(handler, {
+                value: '128',
+                input: {
+                    lambdaARN: 'arnOK',
+                    num: 50,
+                    payloadS3: 's3://my-bucket/my-key.json',
+                    payload: '{"ValueInline": "OK"}', // won't be used
+                },
+            });
+            expect(fetchPayloadFromS3Counter).to.be(1);
+            for (let payload of invokeLambdaPayloads){
+                expect(payload).to.be('{"ValueFromS3":"OK"}');
+            }
+        });
+
+        it('should generate weighted payload from S3', async() => {
+            fetchPayloadFromS3Stub && fetchPayloadFromS3Stub.restore();
+            fetchPayloadFromS3Stub = sandBox.stub(utils, 'fetchPayloadFromS3')
+                .callsFake(async(_arn, _alias, payload) => {
+                    fetchPayloadFromS3Counter++;
+                    return [
+                        { payload: {test: 'A'}, weight: 10 },
+                        { payload: {test: 'B'}, weight: 30 },
+                        { payload: {test: 'C'}, weight: 60 },
+                    ];
+                });
+
+            await invokeForSuccess(handler, {
+                value: '128',
+                input: {
+                    lambdaARN: 'arnOK',
+                    num: 10,
+                    payloadS3: 's3://my-bucket/my-key.json',
+                },
+            });
+
+            expect(fetchPayloadFromS3Counter).to.be(1);
+            expect(invokeLambdaPayloads.length).to.be(10);
+            const counters = {
+                A: 0, B: 0, C: 0,
+            };
+            invokeLambdaPayloads.forEach(payload => {
+                counters[JSON.parse(payload).test] += 1;
+            });
+            expect(counters.A).to.be(1);
+            expect(counters.B).to.be(3);
+            expect(counters.C).to.be(6);
+        });
+
     });
 
     describe('analyzer', () => {
 
         const handler = require('../../lambda/analyzer').handler;
 
-        it('should explode if invoked without invalid event', async() => {
-            const invalidEvents = [
-                null,
-                {},
-                [],
-                { lambdaARN: '' },
-                { whatever: 1 },
-            ];
-            invalidEvents.forEach(event => {
-                expect(async() => {
-                    await invokeForFailure(handler, event);
-                }).to.not.throwError();
+        let invalidEvents = [
+            null,
+            {},
+            [],
+            { lambdaARN: '' },
+            { whatever: 1 },
+        ];
+
+        invalidEvents.forEach(async(event) => {
+            it('should explode if invoked without invalid event - ' + JSON.stringify(event), async() => {
+                await invokeForFailure(handler, event);
             });
         });
 
@@ -1266,19 +1325,30 @@ describe('Lambda Functions', async() => {
             expect(result).to.be(undefined);
         });
 
-        it('should explode if invalid strategy', async() => {
-            const event = {
-                strategy: 'foobar',
-                stats: [
-                    { value: '128', averagePrice: 100, averageDuration: 300, totalCost: 1 },
-                    { value: '256', averagePrice: 200, averageDuration: 200, totalCost: 1 },
-                    { value: '512', averagePrice: 300, averageDuration: 100, totalCost: 1 },
-                ],
-            };
+        const invalidStrategies = [
+            'foobar', // doesn't exist
+            'Balanced', // capitalized
+            ' ',
+            '-',
+        ];
+
+        invalidStrategies.forEach(strategy => {
+
+            it(`should explode if invalid strategy - "${strategy}"`, async() => {
+
+                const event = {
+                    strategy: strategy,
+                    stats: [
+                        { value: '128', averagePrice: 100, averageDuration: 300, totalCost: 1 },
+                        { value: '256', averagePrice: 200, averageDuration: 200, totalCost: 1 },
+                        { value: '512', averagePrice: 300, averageDuration: 100, totalCost: 1 },
+                    ],
+                };
 
-            expect(async() => {
                 await invokeForFailure(handler, event);
-            }).to.not.throwError();
+
+            });
+
         });
 
     });
@@ -1322,23 +1392,22 @@ describe('Lambda Functions', async() => {
                 });
         });
 
-        it('should explode if invoked without lambdaARN or optimal power', async() => {
-            const invalidEvents = [
-                {},
-                { lambdaARN: null },
-                { lambdaARN: '' },
-                { lambdaARN: false },
-                { lambdaARN: 0 },
-                { lambdaARN: '', analysis: null },
-                { lambdaARN: 'arnOK', analysis: {} },
-                { lambdaARN: 'arnOK', analysis: { power: null} },
-            ];
-            invalidEvents.forEach(event => {
-                expect(async() => {
-                    await invokeForFailure(handler, event);
-                }).to.not.throwError();
-            });
 
+        let invalidEvents = [
+            {},
+            { lambdaARN: null },
+            { lambdaARN: '' },
+            { lambdaARN: false },
+            { lambdaARN: 0 },
+            { lambdaARN: '', analysis: null },
+            { lambdaARN: 'arnOK', analysis: {} },
+            { lambdaARN: 'arnOK', analysis: { power: null} },
+        ];
+
+        invalidEvents.forEach(async(event) => {
+            it('should explode if invoked without lambdaARN or optimal power - ' + JSON.stringify(event), async() => {
+                await invokeForFailure(handler, event);
+            });
         });
 
         it('should not do anything if invoked without autoOptimize', async() => {
diff --git a/test/unit/test-utils.js b/test/unit/test-utils.js
index d2d801b..ddb1595 100644
--- a/test/unit/test-utils.js
+++ b/test/unit/test-utils.js
@@ -31,6 +31,7 @@ AWS.mock('Lambda', 'deleteFunction', {});
 AWS.mock('Lambda', 'createAlias', {});
 AWS.mock('Lambda', 'deleteAlias', {});
 AWS.mock('Lambda', 'invoke', {});
+AWS.mock('S3', 'getObject', {Body: Buffer.from('{"Value": "OK"}')});
 
 // note: waiters aren't correctly mocked by aws-sdk-mock (for now)
 // https://github.com/dwyl/aws-sdk-mock/issues/173
@@ -209,7 +210,7 @@ describe('Lambda Utils', () => {
 
     describe('computeAverageDuration', () => {
         const durations = [
-            1, 1, 2, 3, 2000
+            1, 1, 2, 3, 2000,
         ];
 
         it('should return the average duration', () => {
@@ -716,4 +717,107 @@ describe('Lambda Utils', () => {
 
     });
 
+    describe('fetchPayloadFromS3', () => {
+
+        it('should fetch the object from S3 if valid URI', async() => {
+            const payload = await utils.fetchPayloadFromS3('s3://my-bucket/my-key.json');
+            expect(payload).to.be.an('object');
+            expect(payload.Value).to.be('OK');
+        });
+
+        const invalidURIs = [
+            '',
+            '/',
+            'bucket/key.json',
+            '/bucket/key.json',
+            '/key.json',
+            'key.json',
+            's3://bucket/',
+            's3://key.json',
+            's3://',
+        ];
+
+        invalidURIs.forEach(async(uri) => {
+            it(`should explode if invalid URI - ${uri}`, async() => {
+                try {
+                    await utils.fetchPayloadFromS3(uri);
+                    throw new Error(`${uri} did not throw`);
+                } catch (err) {
+                    expect(err.message).to.contain('Invalid S3 path');
+                }
+            });
+        });
+
+        it('should throw if access denied', async() => {
+            AWS.remock('S3', 'getObject', (params, callback) => {
+                const err = new Error('Access Denied');
+                err.statusCode = 403;
+                callback(err, null);
+            });
+            try {
+                await utils.fetchPayloadFromS3('s3://bucket/key.json');
+                throw new Error('Did not catch 403');
+            } catch (err) {
+                expect(err.message).to.contain('Permission denied');
+            }
+        });
+
+        it('should throw if object not found', async() => {
+            AWS.remock('S3', 'getObject', (params, callback) => {
+                const err = new Error('Object not found');
+                err.statusCode = 404;
+                callback(err, null);
+            });
+            try {
+                await utils.fetchPayloadFromS3('s3://bucket/key.json');
+                throw new Error('Did not catch 404');
+            } catch (err) {
+                expect(err.message).to.contain('does not exist');
+            }
+        });
+
+        it('should throw if unknown error', async() => {
+            AWS.remock('S3', 'getObject', (params, callback) => {
+                const err = new Error('Whatever error');
+                err.statusCode = 500;
+                callback(err, null);
+            });
+            try {
+                await utils.fetchPayloadFromS3('s3://bucket/key.json');
+                throw new Error('Did not catch unknown error');
+            } catch (err) {
+                expect(err.message).to.contain('Unknown error');
+                expect(err.message).to.contain('Whatever error');
+            }
+        });
+
+        const validJson = [
+            '{"value": "ok"}',
+            '[1, 2, 3]',
+            '[{"value": "ok"}, {"value2": "ok2"}]',
+        ];
+
+        validJson.forEach(async(str) => {
+            it('should parse string if valid json - ' + str, async() => {
+                AWS.remock('S3', 'getObject', (params, callback) => {
+                    callback(null, {Body: str});
+                });
+
+                const payload = await utils.fetchPayloadFromS3('s3://bucket/key.json');
+                expect(payload).to.be.an('object');
+            });
+        });
+
+        it('should return string if invalid json', async() => {
+            AWS.remock('S3', 'getObject', (params, callback) => {
+                callback(null, {Body: 'just a string'});
+            });
+
+            const payload = await utils.fetchPayloadFromS3('s3://bucket/key.json');
+            expect(payload).to.be.a('string');
+            expect(payload).to.equal('just a string');
+        });
+
+    });
+
 });