Repository: brefphp/bref (language: php)

Issue:
Get rid of `separateVendor: true` for Bref v2
I am considering removing the `separateVendor: true` feature for Bref 2.0 (https://bref.sh/docs/environment/php.html#separate-vendor-folder).

Motivation:

- this feature was contributed a while ago but it's never been maintained, it adds a lot of technical debt and bugs (e.g. last one is #1311)
- I'm not sure it is used a lot
- I'd actually recommend deploying using Docker images instead, as this provides flat cold starts (better performances)

Any opinions here? Anyone using it?


Discussion / hints:
I would say it is not necessary, especially if it slows the whole thing down.

I am still playing around with both bref/serverless and Laravel Vapor.
I had to use `separateVendor: true` otherwise it died with memory error. However playing also with Vapor left me 2 choices - reduce vendor directory or use Docker which I was reluctant to do.
Investigated further, turns out I had a package that ships whole GeoLite database along adding about 50 megs (about 30% of whole vendor dir) I ditched it and Vapor was happy to pack it.

There is also the huge aws package that adds about 35 megs (https://github.com/aws/aws-sdk-php/issues/808), everything else is reasonably sized and does not create any trouble.

```
$ du -h -d 1 vendor/ | sort -h
# lots of small packages
...
7.9M    vendor/symfony
13M     vendor/laravel
37M     vendor/aws
103M    vendor/
```

I tried using it as it allows functions to be updated individually but kept getting 502 errors in production. Didn't have the bandwidth to debug the core issue so I abandoned the idea. I would agree that it it's just adding bugs then it's best not to have it.

Fix patch:
diff --git a/docs/environment/php.md b/docs/environment/php.md
index 6d9ceea43..04f3a9041 100644
--- a/docs/environment/php.md
+++ b/docs/environment/php.md
@@ -177,17 +177,3 @@ provider:
 ```
 
 The path must start with `/var/task`, which is the directory where projects are installed on AWS Lambda.
-
-## Separate vendor folder
-
-Bref has the ability to shrink the Lambda deployment archive by separating the vendor folder from the applications code and injecting it later on into the Lambda function. This allows Bref to circumvent Lambda file size limitations.
-
-By setting the option `separateVendor` in the `custom.bref` block to `true`, you can enable this feature.
-
-```yaml
-custom:
-    bref:
-        separateVendor: true
-```
-
-> Using this option changes the path of the vendor directory to `/tmp/vendor`. References in your code to the composer autoloader therefore must be adjusted to `/tmp/vendor/autoload.php`.
diff --git a/docs/environment/serverless-yml.md b/docs/environment/serverless-yml.md
index 5db7267dc..ffaff970a 100644
--- a/docs/environment/serverless-yml.md
+++ b/docs/environment/serverless-yml.md
@@ -188,8 +188,6 @@ provider:
 
 If you only want to define some permissions **per function**, instead of globally (ie: in the provider), you should install and enable the Serverless plugin [`serverless-iam-roles-per-function`](https://github.com/functionalone/serverless-iam-roles-per-function) and then use the `iamRoleStatements` at the function definition block.
 
-If you are using the custom parameter `separateVendor: true` with **per function** permissions, you should also add the `iamRoleStatementsInherit: true` parameter at the function definition block otherwise the Lambda won't have permissions to retrieve the vendor directory in the S3 bucket anymore.
-
 ## Resources
 
 ```yaml
diff --git a/docs/environment/storage.md b/docs/environment/storage.md
index e5b51f37f..0d3308909 100644
--- a/docs/environment/storage.md
+++ b/docs/environment/storage.md
@@ -4,6 +4,20 @@ current_menu: storage
 introduction: Learn how to store data and files in serverless PHP applications running on AWS Lambda.
 ---
 
+Here is a simplified overview of the filesystem on AWS Lambda:
+
+```bash
+/opt/
+    # Where Lambda layers (like Bref) are unzipped
+/var/task/
+    # Where your application code is unzipped
+/tmp/
+    # Temporary files
+...
+```
+
+The `/var/task` directory is [limited to 250MB](https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html#function-configuration-deployment-and-execution). If you hit that limit, you can deploy [via Docker images instead](../web-apps/docker.md).
+
 The filesystem on AWS Lambda **is read-only**, except for the `/tmp` directory.
 
 On top of that, the filesystem is not shared between instances of a lambda when it scales up. For example a file `/tmp/foo.json` will not be shared across instances of the same lambda. Since a lambda can scale up or down at any time, data in the `/tmp` directory can be lost.
diff --git a/docs/web-apps/docker.md b/docs/web-apps/docker.md
index 03950fcda..c9923b16c 100644
--- a/docs/web-apps/docker.md
+++ b/docs/web-apps/docker.md
@@ -6,12 +6,13 @@ previous:
     title: Local development for web apps
 ---
 
-AWS Lambda support running a Docker Container as a handler
-for your function. This alternative **is a last resort** when:
+AWS Lambda supports running a Docker image, instead of running your application in the default Linux environment. We recommend Docker **as a last resort**, as it is less practical and usually comes with slightly worse cold starts. Yes, Docker is great and probably sounds familiar, but is often not worth it on Lambda.
 
-- Your Lambda Function is larger than 250MB when unzipped
-- You need more than 5 extensions
-- You need resources locally installed (e.g. mysqldump)
+You should consider deploying using Docker when:
+
+- Your Lambda Function is [larger than 250MB when unzipped](../environment/storage.md)
+- You reached the limit of 5 Lambda layers (e.g. for extra PHP extensions)
+- You need resources installed locally (e.g. mysqldump)
 
 > Note: this documentation page assumes that you have read about [web apps on Lambda](../runtimes/http.md) first.
 
diff --git a/index.js b/index.js
index d1ff73bd8..e43179cca 100644
--- a/index.js
+++ b/index.js
@@ -3,6 +3,8 @@
 const {listLayers} = require('./plugin/layers');
 const {runConsole} = require('./plugin/run-console');
 const {runLocal} = require('./plugin/local');
+const fs = require('fs');
+const path = require('path');
 
 /**
  * This file declares a plugin for the Serverless framework.
@@ -13,19 +15,20 @@ const {runLocal} = require('./plugin/local');
 class ServerlessPlugin {
     constructor(serverless, options, utils) {
         this.serverless = serverless;
-        this.options = options;
-        this.utils = utils;
         this.provider = this.serverless.getProvider('aws');
 
-        this.fs = require('fs');
-        this.path = require('path');
-        const filename = this.path.resolve(__dirname, 'layers.json');
-        const layers = JSON.parse(this.fs.readFileSync(filename));
+        if (!utils) {
+            throw new serverless.classes.Error('Bref requires Serverless Framework v3, but an older v2 version is running.\nPlease upgrade to Serverless Framework v3.');
+        }
+
+        const filename = path.resolve(__dirname, 'layers.json');
+        const layers = JSON.parse(fs.readFileSync(filename).toString());
 
         this.checkCompatibleRuntime();
 
         // Declare `${bref:xxx}` variables
         // See https://www.serverless.com/framework/docs/guides/plugins/custom-variables
+        // noinspection JSUnusedGlobalSymbols
         this.configurationVariablesSources = {
             bref: {
                 async resolve({address, resolveConfigurationProperty, options}) {
@@ -55,10 +58,6 @@ class ServerlessPlugin {
             }
         };
 
-        if (!this.utils) {
-            throw new serverless.classes.Error('Bref requires Serverless Framework v3, but an older v2 version is running.\nPlease upgrade to Serverless Framework v3.');
-        }
-
         this.commands = {
             'bref:cli': {
                 usage: 'Runs a CLI command in AWS Lambda',
@@ -100,18 +99,8 @@ class ServerlessPlugin {
             },
         };
 
+        // noinspection JSUnusedGlobalSymbols
         this.hooks = {
-            'initialize': this.addCustomIamRoleForVendorArchiveDownload.bind(this),
-
-            // Separate vendor for `sls deploy` command
-            'package:setupProviderConfiguration': this.createVendorZip.bind(this),
-            'after:aws:deploy:deploy:createStack': this.uploadVendorZip.bind(this),
-            // Separate vendor for `sls deploy function` command
-            'before:deploy:function:initialize': this.createVendorZip.bind(this),
-            'after:deploy:function:initialize': this.uploadVendorZip.bind(this),
-
-            'before:remove:remove': this.removeVendorArchives.bind(this),
-
             // Custom commands
             'bref:cli:run': () => runConsole(this.serverless, options),
             'bref:local:run': () => runLocal(this.serverless, options),
@@ -130,230 +119,6 @@ class ServerlessPlugin {
             }
         }
     }
-
-    addCustomIamRoleForVendorArchiveDownload() {
-        this.serverless.service.custom = this.serverless.service.custom ? this.serverless.service.custom : {};
-        this.serverless.service.custom.bref = this.serverless.service.custom.bref ? this.serverless.service.custom.bref : {};
-        if (! this.serverless.service.custom.bref.separateVendor) {
-            return;
-        }
-
-        this.logVerbose("Adding custom IAM role for vendor archive");
-
-        // If the serverless config does not yet contain an exclude for the vendor folder
-        // we will add it here as we do not want the vendor folder in our
-        // lambda archive file.
-        let excludes = this.serverless.service.package.exclude;
-        if(excludes && excludes.indexOf('vendor/**') === -1) {
-            excludes.push('vendor/**');
-        }
-        let patterns = this.serverless.service.package.patterns;
-        if(patterns && patterns.indexOf('!vendor/**') === -1) {
-            patterns.push('!vendor/**');
-        }
-
-        // This defines the access rights for Lambda, so it can download the
-        // vendor archive file from the vendors subfolder.
-        this.serverless.service.provider.iamRoleStatements = this.serverless.service.provider.iamRoleStatements ? this.serverless.service.provider.iamRoleStatements : [];
-
-        this.serverless.service.provider.iamRoleStatements.push({
-            Effect: 'Allow',
-            Action: [
-                's3:GetObject',
-            ],
-            Resource: [
-                {
-                    'Fn::Join': [
-                        '',
-                        [
-                            'arn:aws:s3:::',
-                            {
-                                'Ref': 'ServerlessDeploymentBucket'
-                            },
-                            '/',
-                            this.stripSlashes(this.provider.getDeploymentPrefix() + '/vendors/*')
-                        ]
-                    ]
-                }
-            ]
-        });
-    }
-
-    async createVendorZip() {
-        this.serverless.service.custom = this.serverless.service.custom ? this.serverless.service.custom : {};
-        this.serverless.service.custom.bref = this.serverless.service.custom.bref ? this.serverless.service.custom.bref : {};
-        if(! this.serverless.service.custom.bref.separateVendor) {
-            return;
-        }
-
-        this.logVerbose("Creating vendor zip file");
-
-        const vendorZipHash = await this.createZipFile();
-        this.newVendorZipName = vendorZipHash + '.zip';
-
-        this.logVerbose('Setting environment variables');
-
-        if (! this.serverless.service.provider.environment) {
-            this.serverless.service.provider.environment = [];
-        }
-
-        // This environment variable will trigger Bref to download the zip on cold start
-        this.serverless.service.provider.environment.BREF_DOWNLOAD_VENDOR = {
-            'Fn::Join': [
-                '',
-                [
-                    's3://',
-                    {
-                        'Ref': 'ServerlessDeploymentBucket'
-                    },
-                    '/',
-                    this.stripSlashes(this.provider.getDeploymentPrefix() + '/vendors/' + this.newVendorZipName)
-                ]
-            ]
-        };
-    }
-
-    async createZipFile() {
-        const vendorDir = '.serverless';
-        if (!this.fs.existsSync(vendorDir)){
-            this.fs.mkdirSync(vendorDir);
-        }
-        this.filePath = `${vendorDir}/vendor.zip`;
-
-        return await new Promise((resolve, reject) => {
-            const archiver = require(process.mainModule.path + '/../node_modules/archiver');
-            const output = this.fs.createWriteStream(this.filePath);
-            const archive = archiver('zip', {
-                zlib: { level: 9 } // Highest compression level.
-            });
-
-            this.logVerbose(`Packaging the Composer vendor directory in ${this.filePath}`);
-
-            archive.pipe(output);
-            archive.directory('vendor/', false);
-            archive.finalize();
-
-            output.on('close', () => {
-                this.logVerbose(`Created vendor.zip with ${archive.pointer()} total bytes.`);
-                resolve();
-            });
-
-            archive.on('warning', err => {
-                if (err.code === 'ENOENT') {
-                    this.utils.log.warning('Bref: Archiver warning: ' + err);
-                } else {
-                    throw new Error(err);
-                }
-            });
-
-            archive.on('error', err => {
-                throw new Error(err);
-            });
-        })
-            .then(() => {
-                // We will rename vendor.zip to a unique name to:
-                // - avoid overwriting zips from previous deployments running in production
-                // - avoid deploying vendor.zip with exactly the same contents
-                const crypto = require('crypto');
-
-                return new Promise(resolve => {
-                    const hash = crypto.createHash('md5');
-                    this.fs.createReadStream(this.filePath).on('data', data => hash.update(data)).on('end', () => resolve(hash.digest('hex')));
-                });
-            })
-            .catch(err => {
-                throw new Error(`Failed to create zip file vendor.zip: ${err.message}`);
-            });
-    }
-
-    async uploadVendorZip() {
-        this.serverless.service.custom = this.serverless.service.custom ? this.serverless.service.custom : {};
-        this.serverless.service.custom.bref = this.serverless.service.custom.bref ? this.serverless.service.custom.bref : {};
-        if(! this.serverless.service.custom.bref.separateVendor) {
-            return;
-        }
-
-        this.logVerbose("Uploading vendor zip file...");
-
-        await this.uploadZipToS3(this.filePath);
-
-        this.logVerbose('Vendor separation done');
-    }
-
-    async uploadZipToS3(zipFile) {
-        const bucketName = await this.provider.getServerlessDeploymentBucketName();
-        const deploymentPrefix = await this.provider.getDeploymentPrefix();
-
-        this.logVerbose('Checking vendor file on bucket...');
-
-        try {
-            await this.provider.request('S3', 'headObject', {
-                Bucket: bucketName,
-                Key: this.stripSlashes(deploymentPrefix + '/vendors/' + this.newVendorZipName)
-            });
-
-            this.logVerbose('Vendor file already exists on bucket. Not uploading again.');
-            return;
-        } catch(e) {
-            // The vendor file needs to be uploaded
-            this.logVerbose('Vendor file not found. Uploading...');
-        }
-
-        const readStream = this.fs.createReadStream(zipFile);
-        const details = {
-            ACL: 'private',
-            Body: readStream,
-            Bucket: bucketName,
-            ContentType: 'application/zip',
-            Key: this.stripSlashes(deploymentPrefix + '/vendors/' + this.newVendorZipName),
-        };
-
-        return await this.provider.request('S3', 'putObject', details);
-    }
-
-    stripSlashes(path) {
-        return path.replace(/^\/+/g, '');
-    }
-
-    /**
-     * CloudFormation cannot delete a bucket that contains files.
-     * That's why we clean up the vendor zip files when `serverless remove` is being run.
-     */
-    async removeVendorArchives() {
-        const bucketName = await this.provider.getServerlessDeploymentBucketName();
-        const deploymentPrefix = await this.provider.getDeploymentPrefix();
-
-        const bucketObjects = await this.provider.request('S3', 'listObjectsV2', {
-            Bucket: bucketName,
-            Prefix: this.stripSlashes(deploymentPrefix + '/vendors/')
-        });
-        if (bucketObjects.Contents.length === 0) {
-            return;
-        }
-
-        this.logVerbose('Removing Composer `vendor` archives from the S3 bucket.');
-
-        let details = {
-            Bucket: bucketName,
-            Delete: {
-                Objects: []
-            }
-        };
-
-        bucketObjects.Contents.forEach(content => {
-            details.Delete.Objects.push({
-                Key: content.Key
-            });
-        });
-
-        this.logVerbose(`Found ${details.Delete.Objects.length} vendor archives. Removing them from Bucket now.`);
-
-        return await this.provider.request('S3', 'deleteObjects', details);
-    }
-
-    logVerbose(message) {
-        this.utils.log.verbose(`Bref: ${message}`);
-    }
 }
 
 module.exports = ServerlessPlugin;


Test patch: