Repository: serverless/serverless (JavaScript)
Date: 2019-09-09 12:09:52+00:00
Task type: Bug Fix

Issue:
Deploy stops at "Installing dependencies for custom CloudFormation resources"
I´m using this template https://github.com/msfidelis/serverless-architecture-boilerplate
and the issue has only occured with this template so far.

Creating new projects from scratch with the sls assistent and deploying works fine.

When running **sls deploy** it processes these 3 steps and then stops

```
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Installing dependencies for custom CloudFormation resources...
```

Running deploy with **--verbose --force** does not change anything and there are literally no additional exceptions or log messages thrown which makes it very hard to find the source of this issue.

Serverless v: 1.50.0
OS: Windows 10


Issue discussion:
I also experienced the same problem, but with the error message: 

`EMFILE: too many open files, open 'C: \ Users \ marda \ AppData \ Local \ Temp \ tmpdirs-serverless \ 330d \ b435695edb9e63a1 \ resources \ node_modules \ tokopedia-api \ api node_modules \ caniuse-lite \ data \ regions \ GL.js'`

```
 Your Environment Information ---------------------------
     Operating System:          win32
     Node Version:              12.8.1
     Framework Version:         1.50.0
     Plugin Version:            1.3.8
     SDK Version:               2.1.0
```

Serverless v: 1.50.0
OS: Windows 10

Update: 
After I tried to downgrade to the previous version (v1.49.0) it turns out that this message disappears and serverless runs normally. I think this is a bug for version v1.50.0
@mklueh Great thanks for reporting, we'll look into that
> I also experienced the same problem, but with the error message:
> 
> `EMFILE: too many open files, open 'C: \ Users \ marda \ AppData \ Local \ Temp \ tmpdirs-serverless \ 330d \ b435695edb9e63a1 \ resources \ node_modules \ tokopedia-api \ api node_modules \ caniuse-lite \ data \ regions \ GL.js'`
> 
> ```
>  Your Environment Information ---------------------------
>      Operating System:          win32
>      Node Version:              12.8.1
>      Framework Version:         1.50.0
>      Plugin Version:            1.3.8
>      SDK Version:               2.1.0
> ```
> 
> Serverless v: 1.50.0
> OS: Windows 10
> 
> Update:
> After I tried to downgrade to the previous version (v1.49.0) it turns out that this message disappears and serverless runs normally. I think this is a bug for version v1.50.0

I'm getting a similar error with `EMFILE: too many open files`, also on win32 and serverless 1.50.0.
Unfortunately downgrading to 1.49.0 didnt help for me, but downgrade to 1.48.4 did remove the issue.
Same here on version 1.50.0 with the same EMFILE error as @mvrska, downgrading to 1.48.4 also worked for me. Windows 10 Version 10.0.18362 Build 18362.

Chiming in, same story, 1.50.0 too many files, 1.48.4 deploys just fine. 


```  Your Environment Information ---------------------------
     Operating System:          win32
     Node Version:              10.16.3
     Framework Version:         1.50.0
     Plugin Version:            1.3.8
     SDK Version:               2.1.0```
@dested Downgrading to 1.48.4 has indeed worked for me too. Thank you
This happens to me to, however i cannot downgrade below 1.49 as I wish to use eventBridge. Any suggestions would be great!

`  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information ---------------------------
     Operating System:          win32
     Node Version:              12.9.0
     Framework Version:         1.51.0
     Plugin Version:            1.3.9
     SDK Version:               2.1.0`
1.51.0 still has the same error.
Did they even try to fix the bug before the 1.51 release?
Looks to only affect windows so likely they aren't being affected thus making it less of a priority
Work to be done is scheduled here: https://github.com/serverless/serverless/issues/6598.
It's in a queue, and if not with next minor release, I believe it should land with following one.
Hey everyone,
thanks for reporting and confirming this issue. I just created https://github.com/serverless/serverless/issues/6557 to explore different solutions for this issue.

Unfortunately I don't have a Windows machine handy, so it would be super awesome if someone can checkout this PR and report back if this fixes the problem.

This issue seems to be related to https://github.com/serverless/serverless/issues/3249.

---

**EDIT:** apparently the current fix doesn't do anything at all. I took another deep dive into this and it seems like the zipping is the issue here (see: https://github.com/serverless/serverless/pull/6636#issuecomment-528360012). I'll work on an update on that front and post another comment when I have something to test.
Hey everyone,
here's just a quick update: I've pushed a fix in #6636 in which I unified our zipping approaches. We're now using one zip util function for both, custom resource zipping and service zipping. This implementation uses `graceful-fs` when working with files in an async fashion so that the aforementioned bugs in Windows shouldn't happen anymore.

Would be really awesome if someone can test this branch out and confirm that this fixes the problem (I don't have a Windows machine). Thanks in advance!
@pmuens Just tested: the fix works great on Windows. Would be nice if you could publish a new version soon.
For those who need the fix right now, here is a diff which can be applied using [patch-package](https://github.com/ds300/patch-package): [github gist](https://gist.github.com/hakimio/01e51dfda185094ba4806f460d425ec3)

BTW, @pmuens is there any reason you can't just create a VM on Azure to test fixes for Windows?


Solution patch:
diff --git a/bin/serverless.js b/bin/serverless.js
index 82bc47a1c54..77d4a0150c9 100755
--- a/bin/serverless.js
+++ b/bin/serverless.js
@@ -2,6 +2,13 @@
 
 'use strict';
 
+// global graceful-fs patch
+// https://github.com/isaacs/node-graceful-fs#global-patching
+const realFs = require('fs');
+const gracefulFs = require('graceful-fs');
+
+gracefulFs.gracefulify(realFs);
+
 const userNodeVersion = Number(process.version.split('.')[0].slice(1));
 
 // only check for components if user is running Node 8
diff --git a/lib/plugins/package/lib/zipService.js b/lib/plugins/package/lib/zipService.js
index 739d40b9b79..d9eca72bb2f 100644
--- a/lib/plugins/package/lib/zipService.js
+++ b/lib/plugins/package/lib/zipService.js
@@ -5,7 +5,7 @@ const archiver = require('archiver');
 const os = require('os');
 const path = require('path');
 const crypto = require('crypto');
-const fs = BbPromise.promisifyAll(require('graceful-fs'));
+const fs = BbPromise.promisifyAll(require('fs'));
 const childProcess = BbPromise.promisifyAll(require('child_process'));
 const globby = require('globby');
 const _ = require('lodash');


Test patch:
diff --git a/lib/plugins/package/lib/zipService.test.js b/lib/plugins/package/lib/zipService.test.js
index b7050dd6416..f8b964a488b 100644
--- a/lib/plugins/package/lib/zipService.test.js
+++ b/lib/plugins/package/lib/zipService.test.js
@@ -9,7 +9,7 @@ const JsZip = require('jszip');
 const globby = require('globby');
 const _ = require('lodash');
 const BbPromise = require('bluebird');
-const fs = BbPromise.promisifyAll(require('graceful-fs'));
+const fs = BbPromise.promisifyAll(require('fs'));
 const childProcess = BbPromise.promisifyAll(require('child_process'));
 const sinon = require('sinon');
 const Package = require('../package');