Repository: iterative/dvc
Pull request: remote: s3: adjust jobs number basing on file descriptors number
* [x] ❗ Have you followed the guidelines in the [Contributing to DVC](https://dvc.org/doc/user-guide/contributing/core) list?

* [ ] 📖 Check this box if this PR **does not** require [documentation](https://dvc.org/doc) updates, or if it does **and** you have created a separate PR in [dvc.org](https://github.com/iterative/dvc.org) with such updates (or at least opened an issue about it in that repo). Please link below to your PR (or issue) in the [dvc.org](https://github.com/iterative/dvc.org) repo.

* [x] ❌ Have you checked DeepSource, CodeClimate, and other sanity checks below? We consider their findings recommendatory and don't expect everything to be addressed. Please review them carefully and fix those that actually improve code or fix bugs.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

Fixes #2473 

Problem statement:
'Errno 24 - Too many open files' on dvc push
### Version information
* DVC version: 0.58.1
* Platform: MacOS 10.14.6
* Method of installation: pip within a conda environment

### Description
When pushing to S3 a directory of ~100 files that have been added to DVC, I observe an Errno 24 error from the dvc process.

It looks like dvc is trying to open more files than the OS allows. Checking the file handles on for the dvc process I get:

```
$ lsof -p $DVC_PID | wc -l
412
```

Looking at the OS limits, a process is limited to having 256 open files.

```
$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4256
virtual memory          (kbytes, -v) unlimited
```

A workaround for this is to increase the max files per process to a larger number (say 4096) by running something like `ulimit -n 4096`, but I wonder if the ideal solution is for DVC to work within the OS configured limits by default?

Edit: Updated wording of workaround

Hints from issue discussion:
Hi @ChrisHowlin !

Thank you for reporting this! :slightly_smiling_face: 

Could you post full log, please?

It is 100 files, right? Not 100K? Just making sure I understand you correctly. If it is 100, we might be leaking fds :slightly_frowning_face: Mind also trying dvc version 0.56.0, to see if that work? We've had some changes to dvc since that version, that I suspect might be the cause of the leaks.

Thanks,
Ruslan

I checked again, and the number of files being pushed was actually 79 (definitely not thousands 😄 )

I don't have the full logs from the `dvc push` to hand right now, but I was getting the below for repeatedly for various files. This seemed to be happening as soon as the first files finished uploading.

```
ERROR: failed to upload '.dvc/cache/2f/25387c98c599ab0de148f437b780ad' to 's3://<s3_path>/2f/25387c98c599ab0de148f437b780ad' - [Errno 24] Too many open files: '/<local_path>/.dvc/cache/2f/25387c98c599ab0de148f437b780ad'
Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
```

I don't have access to the environment until Monday, but I might be able to reproduce over the weekend.
Hi @ChrisHowlin ! Were you able to reproduce it again? :slightly_smiling_face: 
Just for the record, we are doing tests with millions of files and not seeing reaching the linux opened fds limit (which is 4K), so I don't think we are seriously leaking fds. But with 256 on mac and 70 files, in 4*NCPU workers, we might indeed be pushing it a bit. Using lower `--jobs` number should make this work, but we might want to take a look into our opened fds just to see if there is anything we can close without harm. Would love to hear if that workaround actually works for you, @ChrisHowlin . :slightly_smiling_face:   
I have got some time to have a look again today, and going through the steps to try and reproduce with 0.56.0.

Some observations I have so far is that, this error appears right at the start of the push, before I see any progress on the upload bars. Also, tracking the lsof count for the dvc push process over time shows the number of open files reducing over time.

I wonder if the problem is not an fd leak, instead too many threads are opening files and sockets to S3, beyond the limit the OS defines for the process (default 256)?

I will provide some more logs on the different versions when I have them.

I have managed to reproduce on 0.56.0 and also have identified minimal-ish reproduction instructions using test data:

### Repro instructions
```

# Pre-requisites
pip install dvc==0.56.0
ulimit -n 256  # Set open files per proc limit to 256, if not set already

# Create new repo
mkdir dvc_many_files
cd dvc_many_files
git init
dvc init
dvc remote add -d upstream s3://<S3 URL>
mkdir data

# Create 100 files of 100MB using random data
# Note, this is potentially bash and MacOS specific
for i in {1..100}; do dd if=/dev/urandom bs=100000000 count=1 of=data/file$i; done

dvc add -R data

# Error occurs here
dvc push
```

### Repro log for `dvc push` (on 0.56.0)
```
~/dvc_many_files_test $ dvc push
+-------------------------------------------+
|                                           |
|     Update available 0.56.0 -> 0.59.2     |
|       Run pip install dvc --upgrade       |
|                                           |
+-------------------------------------------+

Preparing to upload data to 's3://<S3 bucket>/dvc_many_files_test'
Preparing to collect status from s3://<S3 bucket>/dvc_many_files_test
Collecting information from local cache...
[##############################] 100%

Collecting information from remote cache...
[##############################] 100%
[##############################] 100% Analysing status
[                              ] 2% data/file91
ERROR: failed to upload '.dvc/cache/94/f505e87c2ecf8800c18ef183e5fc1f' to 's3://<S3 bucket>/dvc_many_files_test/94/f505e87c2ecf8800c18ef183e5fc1f' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/94/f505e87c2ecf8800c18ef183e5fc1f'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[#                             ] 6% data/file53
ERROR: failed to upload '.dvc/cache/28/33a50139e861ababfaae379ee1dfc6' to 's3://<S3 bucket>/dvc_many_files_test/28/33a50139e861ababfaae379ee1dfc6' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/28/33a50139e861ababfaae379ee1dfc6'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] 2% data/file90
[#                             ] 4% data/file47ERROR: failed to upload '.dvc/cache/36/6c860189e1c1bff2d8946c30568a8f' to 's3://<S3 bucket>/dvc_many_files_test/36/6c860189e1c1bff2d8946c30568a8f' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/36/6c860189e1c1bff2d8946c30568a8f'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[#                             ] 4% data/file91
ERROR: failed to upload '.dvc/cache/a2/f1468aeac80f884693b9b208ab01cd' to 's3://<S3 bucket>/dvc_many_files_test/a2/f1468aeac80f884693b9b208ab01cd' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/a2/f1468aeac80f884693b9b208ab01cd'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] ?% data/file83
ERROR: failed to upload '.dvc/cache/ca/13773b5d9feea4a81b9489c7a47a3d' to 's3://<S3 bucket>/dvc_many_files_test/ca/13773b5d9feea4a81b9489c7a47a3d' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/ca/13773b5d9feea4a81b9489c7a47a3d'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[#                             ] 4% data/file91
ERROR: failed to upload '.dvc/cache/e4/1dee6ea60377e6a84c26a830a67360' to 's3://<S3 bucket>/dvc_many_files_test/e4/1dee6ea60377e6a84c26a830a67360' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/e4/1dee6ea60377e6a84c26a830a67360'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] 1% data/file84
ERROR: failed to upload '.dvc/cache/90/809bb6e9bcb883ac15de437293c989' to 's3://<S3 bucket>/dvc_many_files_test/90/809bb6e9bcb883ac15de437293c989' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/90/809bb6e9bcb883ac15de437293c989'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[#                             ] 5% data/file47
ERROR: failed to upload '.dvc/cache/52/09af8f9845b5f9c24d1db75d17d85a' to 's3://<S3 bucket>/dvc_many_files_test/52/09af8f9845b5f9c24d1db75d17d85a' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/52/09af8f9845b5f9c24d1db75d17d85a'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[#                             ] 5% data/file91
ERROR: failed to upload '.dvc/cache/03/5b32c997321d49f64f0783b66f1e4e' to 's3://<S3 bucket>/dvc_many_files_test/03/5b32c997321d49f64f0783b66f1e4e' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/03/5b32c997321d49f64f0783b66f1e4e'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[#                             ] 6% data/file52
ERROR: failed to upload '.dvc/cache/ab/898d5a907d0f7c44670d4fb3cf200c' to 's3://<S3 bucket>/dvc_many_files_test/ab/898d5a907d0f7c44670d4fb3cf200c' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/ab/898d5a907d0f7c44670d4fb3cf200c'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] 2% data/file84
ERROR: failed to upload '.dvc/cache/bc/ad450a109af254c8be7790b19c5a69' to 's3://<S3 bucket>/dvc_many_files_test/bc/ad450a109af254c8be7790b19c5a69' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/bc/ad450a109af254c8be7790b19c5a69'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] 0% data/file97
ERROR: failed to upload '.dvc/cache/3f/0d47fdc3cc700aea48e48a8ded4c5c' to 's3://<S3 bucket>/dvc_many_files_test/3f/0d47fdc3cc700aea48e48a8ded4c5c' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/3f/0d47fdc3cc700aea48e48a8ded4c5c'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[#                             ] 6% data/file52
ERROR: failed to upload '.dvc/cache/2e/b4611efe25405c858da0f62fab3c02' to 's3://<S3 bucket>/dvc_many_files_test/2e/b4611efe25405c858da0f62fab3c02' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/2e/b4611efe25405c858da0f62fab3c02'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] 2% data/file50
ERROR: failed to upload '.dvc/cache/5d/6c3144597973b79ea06b368d4b4b24' to 's3://<S3 bucket>/dvc_many_files_test/5d/6c3144597973b79ea06b368d4b4b24' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/5d/6c3144597973b79ea06b368d4b4b24'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[#                             ] 5% data/file51
ERROR: failed to upload '.dvc/cache/4e/7f0799b7972ea0ef8f7edd5816acaa' to 's3://<S3 bucket>/dvc_many_files_test/4e/7f0799b7972ea0ef8f7edd5816acaa' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/4e/7f0799b7972ea0ef8f7edd5816acaa'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] 0% data/file97
ERROR: failed to upload '.dvc/cache/ec/e09f9f079b6b52710490f8a125a431' to 's3://<S3 bucket>/dvc_many_files_test/ec/e09f9f079b6b52710490f8a125a431' - Could not connect to the endpoint URL: "https://<S3 bucket>.s3.eu-west-2.amazonaws.com/dvc_many_files_test/ec/e09f9f079b6b52710490f8a125a431?uploadId=zj5.u8YBvrFL7TXZI6ObT8BdosYfHSOPPyXRndYX0i2rUo.Uhy4jMs7D9E9QX6Hk68cZWl1il_OaZ5Fq11lpojAvsL1Ca.ql4epwIeOHEFrj.IP_IAwNT4BKot.uwQyu&partNumber=9"

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[#                             ] 5% data/file47
ERROR: failed to upload '.dvc/cache/a3/ad651500cc5d1bae786c1658eb06be' to 's3://<S3 bucket>/dvc_many_files_test/a3/ad651500cc5d1bae786c1658eb06be' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/a3/ad651500cc5d1bae786c1658eb06be'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] 1% data/file97
ERROR: failed to upload '.dvc/cache/23/58acefc522fa7585a1fba6e7c35236' to 's3://<S3 bucket>/dvc_many_files_test/23/58acefc522fa7585a1fba6e7c35236' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/23/58acefc522fa7585a1fba6e7c35236'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] 2% data/file86
ERROR: failed to upload '.dvc/cache/2a/1ed0ceca6128c1e55d28b363bded5c' to 's3://<S3 bucket>/dvc_many_files_test/2a/1ed0ceca6128c1e55d28b363bded5c' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/2a/1ed0ceca6128c1e55d28b363bded5c'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] 2% data/file92
ERROR: failed to upload '.dvc/cache/f9/d8068ef642c1d61993151a94fa7523' to 's3://<S3 bucket>/dvc_many_files_test/f9/d8068ef642c1d61993151a94fa7523' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/f9/d8068ef642c1d61993151a94fa7523'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] 2% data/file92
ERROR: failed to upload '.dvc/cache/d4/7a887106ef626f204fc693395a3c5e' to 's3://<S3 bucket>/dvc_many_files_test/d4/7a887106ef626f204fc693395a3c5e' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/d4/7a887106ef626f204fc693395a3c5e'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] 3% data/file50
ERROR: failed to upload '.dvc/cache/1a/bd64ea4334fc8d441236d8a98dd681' to 's3://<S3 bucket>/dvc_many_files_test/1a/bd64ea4334fc8d441236d8a98dd681' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/1a/bd64ea4334fc8d441236d8a98dd681'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] ?% data/file19
ERROR: failed to upload '.dvc/cache/f4/1153ee84d4ce38833cded1f93537f9' to 's3://<S3 bucket>/dvc_many_files_test/f4/1153ee84d4ce38833cded1f93537f9' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/f4/1153ee84d4ce38833cded1f93537f9'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[##                            ] 7% data/file52
ERROR: failed to upload '.dvc/cache/92/6844cee8fbb66f66068a7d86ed770e' to 's3://<S3 bucket>/dvc_many_files_test/92/6844cee8fbb66f66068a7d86ed770e' - Could not connect to the endpoint URL: "https://<S3 bucket>.s3.eu-west-2.amazonaws.com/dvc_many_files_test/92/6844cee8fbb66f66068a7d86ed770e?uploadId=8pDWKXaDGMNhpfCGAl6mekLcuytP6DO15rTla8Gj8pYZQOB2WgVZ8eq7ht8XboEz6IDu8W6sFxqB4hiPRkaYk1XAgDaub.weKDcWcq4XUf84n0i43ap6pPjje057ID5.&partNumber=4"

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[#                             ] 6% data/file47
ERROR: failed to upload '.dvc/cache/7f/0c2f36eed59d8cea3a456368c6fecf' to 's3://<S3 bucket>/dvc_many_files_test/7f/0c2f36eed59d8cea3a456368c6fecf' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/7f/0c2f36eed59d8cea3a456368c6fecf'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[##                            ] 9% data/file51
ERROR: failed to upload '.dvc/cache/d4/3b4c5df21af285f054a611f4595dd2' to 's3://<S3 bucket>/dvc_many_files_test/d4/3b4c5df21af285f054a611f4595dd2' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/d4/3b4c5df21af285f054a611f4595dd2'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[###                           ] 11% data/file47
ERROR: failed to upload '.dvc/cache/50/9c097cf2184038c2f316c02f859ba5' to 's3://<S3 bucket>/dvc_many_files_test/50/9c097cf2184038c2f316c02f859ba5' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/50/9c097cf2184038c2f316c02f859ba5'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[###                           ] 10% data/file91
ERROR: failed to upload '.dvc/cache/f4/2e27370514dddd2d4d6025daa94cea' to 's3://<S3 bucket>/dvc_many_files_test/f4/2e27370514dddd2d4d6025daa94cea' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/f4/2e27370514dddd2d4d6025daa94cea'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] ?% data/file2
ERROR: failed to upload '.dvc/cache/5b/dfda2f9ecaa54927016cbd3d410620' to 's3://<S3 bucket>/dvc_many_files_test/5b/dfda2f9ecaa54927016cbd3d410620' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/5b/dfda2f9ecaa54927016cbd3d410620'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] 2% data/file24
ERROR: failed to upload '.dvc/cache/1c/75b63f844cdde39ab9aeaae0a26425' to 's3://<S3 bucket>/dvc_many_files_test/1c/75b63f844cdde39ab9aeaae0a26425' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/1c/75b63f844cdde39ab9aeaae0a26425'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[##                            ] 8% data/file86
ERROR: failed to upload '.dvc/cache/38/bed7aec1fbba7bc22c11e8c870e93b' to 's3://<S3 bucket>/dvc_many_files_test/38/bed7aec1fbba7bc22c11e8c870e93b' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/38/bed7aec1fbba7bc22c11e8c870e93b'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[###                           ] 10% data/file97
ERROR: failed to upload '.dvc/cache/7e/d55d70e886096b4bff8660244adf4d' to 's3://<S3 bucket>/dvc_many_files_test/7e/d55d70e886096b4bff8660244adf4d' - Could not connect to the endpoint URL: "https://<S3 bucket>.s3.eu-west-2.amazonaws.com/dvc_many_files_test/7e/d55d70e886096b4bff8660244adf4d?uploadId=pd0i0Ja49sxE4J7kMTdAPN37ZSiXuhve6birpZmPDSGIAj4fwIsxlXBk3H4ocURh_4jlh_WjsA1FfoZZ_Hl3_wcOKsvRhrdsfWrdeiuFUhoM0wzIT4Q78Mn3fMj7_RZo&partNumber=6"

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[###                           ] 12% data/file91
ERROR: failed to upload '.dvc/cache/90/93e1dfad3c5f259eb85ec02a2b98e7' to 's3://<S3 bucket>/dvc_many_files_test/90/93e1dfad3c5f259eb85ec02a2b98e7' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/90/93e1dfad3c5f259eb85ec02a2b98e7'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[###                           ] 13% data/file53
[###                           ] 10% data/file97ERROR: failed to upload '.dvc/cache/13/1f00077ff1ebfe7a7203273d82490a' to 's3://<S3 bucket>/dvc_many_files_test/13/1f00077ff1ebfe7a7203273d82490a' - Could not connect to the endpoint URL: "https://<S3 bucket>.s3.eu-west-2.amazonaws.com/dvc_many_files_test/13/1f00077ff1ebfe7a7203273d82490a?uploadId=zR6vIoO0S5l11GnLhU0BLkuheJmP9zKyZhnJLB5Ej1.Tco7hExVUF1mhV3JdTGtP5G9zkkBLgMlNpXAy5qXkclwn7xrhZ9iG4Xpboer.d6GQqevCs5joLwSKVdRFjNkg&partNumber=3"

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[####                          ] 14% data/file53
ERROR: failed to upload '.dvc/cache/7c/ee4d6732d838387c3da0721d60c664' to 's3://<S3 bucket>/dvc_many_files_test/7c/ee4d6732d838387c3da0721d60c664' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/7c/ee4d6732d838387c3da0721d60c664'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] 3% data/file26
ERROR: failed to upload '.dvc/cache/07/8b99f1a3ecb976100f5ff8d9759668' to 's3://<S3 bucket>/dvc_many_files_test/07/8b99f1a3ecb976100f5ff8d9759668' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/07/8b99f1a3ecb976100f5ff8d9759668'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[####                          ] 16% data/file85
ERROR: failed to upload '.dvc/cache/39/66cd187c1450f9a3f4b37f138368d4' to 's3://<S3 bucket>/dvc_many_files_test/39/66cd187c1450f9a3f4b37f138368d4' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/39/66cd187c1450f9a3f4b37f138368d4'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[#                             ] 4% data/file26
ERROR: failed to upload '.dvc/cache/e5/3884fa1cf4a6a25fb774634a24013d' to 's3://<S3 bucket>/dvc_many_files_test/e5/3884fa1cf4a6a25fb774634a24013d' - Could not connect to the endpoint URL: "https://<S3 bucket>.s3.eu-west-2.amazonaws.com/dvc_many_files_test/e5/3884fa1cf4a6a25fb774634a24013d?uploadId=Jc_bEutZbJB2JcnvI2gbP9k4VHgzSnWxFiWpQFqtZURBEmvZLYutM6Edtt.VPJN0nubp6KUtQ4HOl0NJ5qM2mzRl5TQFbgKbHSYgowzQStENYxT9PJHndwqAQQ8osjqE&partNumber=4"

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] ?% data/file36
ERROR: failed to upload '.dvc/cache/04/46e91089131693ff107a0ecfae7ba8' to 's3://<S3 bucket>/dvc_many_files_test/04/46e91089131693ff107a0ecfae7ba8' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/04/46e91089131693ff107a0ecfae7ba8'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[#####                         ] 17% data/file85
ERROR: failed to upload '.dvc/cache/af/af499a3df76772e0b613e2f764ee58' to 's3://<S3 bucket>/dvc_many_files_test/af/af499a3df76772e0b613e2f764ee58' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/af/af499a3df76772e0b613e2f764ee58'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] ?% data/file4
ERROR: failed to upload '.dvc/cache/fc/3d51d2deb8fb8c720fda7442dbe9fe' to 's3://<S3 bucket>/dvc_many_files_test/fc/3d51d2deb8fb8c720fda7442dbe9fe' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/fc/3d51d2deb8fb8c720fda7442dbe9fe'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[####                          ] 14% data/file52
ERROR: failed to upload '.dvc/cache/f4/6366c801ac582633d0861a750a25be' to 's3://<S3 bucket>/dvc_many_files_test/f4/6366c801ac582633d0861a750a25be' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/f4/6366c801ac582633d0861a750a25be'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[###                           ] 13% data/file91
ERROR: failed to upload '.dvc/cache/bc/b0d5fd5cdded766d128fae9c32ff4c' to 's3://<S3 bucket>/dvc_many_files_test/bc/b0d5fd5cdded766d128fae9c32ff4c' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/bc/b0d5fd5cdded766d128fae9c32ff4c'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] ?% data/file34
ERROR: failed to upload '.dvc/cache/e0/ba85929d48d0a150f3f6d3b181ca12' to 's3://<S3 bucket>/dvc_many_files_test/e0/ba85929d48d0a150f3f6d3b181ca12' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/e0/ba85929d48d0a150f3f6d3b181ca12'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[####                          ] 15% data/file52
ERROR: failed to upload '.dvc/cache/f0/0d19e3294331d7254284fd755cdbc9' to 's3://<S3 bucket>/dvc_many_files_test/f0/0d19e3294331d7254284fd755cdbc9' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/f0/0d19e3294331d7254284fd755cdbc9'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[####                          ] 14% data/file51
ERROR: failed to upload '.dvc/cache/40/50627d79382c3abab8ac4436021aa8' to 's3://<S3 bucket>/dvc_many_files_test/40/50627d79382c3abab8ac4436021aa8' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/40/50627d79382c3abab8ac4436021aa8'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] ?% data/file7
ERROR: failed to upload '.dvc/cache/67/b94a63dd146a962feb6944ff775b18' to 's3://<S3 bucket>/dvc_many_files_test/67/b94a63dd146a962feb6944ff775b18' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/67/b94a63dd146a962feb6944ff775b18'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[###                           ] 13% data/file91
ERROR: failed to upload '.dvc/cache/51/a986ed06d5ea6eea398e48d9a7f2b5' to 's3://<S3 bucket>/dvc_many_files_test/51/a986ed06d5ea6eea398e48d9a7f2b5' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/51/a986ed06d5ea6eea398e48d9a7f2b5'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[###                           ] 10% data/file86
ERROR: failed to upload '.dvc/cache/88/d82ec856e5d0c1e2f8c3422f32e111' to 's3://<S3 bucket>/dvc_many_files_test/88/d82ec856e5d0c1e2f8c3422f32e111' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/88/d82ec856e5d0c1e2f8c3422f32e111'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[#                             ] 5% data/file26
ERROR: failed to upload '.dvc/cache/9e/0afa862975614304a5fefe07ec2f14' to 's3://<S3 bucket>/dvc_many_files_test/9e/0afa862975614304a5fefe07ec2f14' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/9e/0afa862975614304a5fefe07ec2f14'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[#                             ] 5% data/file26
ERROR: failed to upload '.dvc/cache/75/194fd4bbc5d039802bb801e297dbf4' to 's3://<S3 bucket>/dvc_many_files_test/75/194fd4bbc5d039802bb801e297dbf4' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/75/194fd4bbc5d039802bb801e297dbf4'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[###                           ] 11% data/file43
ERROR: failed to upload '.dvc/cache/a5/344804c085d536e8d104e41533c57d' to 's3://<S3 bucket>/dvc_many_files_test/a5/344804c085d536e8d104e41533c57d' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/a5/344804c085d536e8d104e41533c57d'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[####                          ] 14% data/file91
ERROR: failed to upload '.dvc/cache/7d/59046fed8e557a41ec8facf49e2d21' to 's3://<S3 bucket>/dvc_many_files_test/7d/59046fed8e557a41ec8facf49e2d21' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/7d/59046fed8e557a41ec8facf49e2d21'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[#                             ] 6% data/file26
ERROR: failed to upload '.dvc/cache/67/49e8b469a645f927569c00952da2d6' to 's3://<S3 bucket>/dvc_many_files_test/67/49e8b469a645f927569c00952da2d6' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/67/49e8b469a645f927569c00952da2d6'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[#                             ] 6% data/file26
ERROR: failed to upload '.dvc/cache/61/5e6683ddfbbf81102d63c6ba796269' to 's3://<S3 bucket>/dvc_many_files_test/61/5e6683ddfbbf81102d63c6ba796269' - Could not connect to the endpoint URL: "https://<S3 bucket>.s3.eu-west-2.amazonaws.com/dvc_many_files_test/61/5e6683ddfbbf81102d63c6ba796269?uploadId=Jra18INvz_FIo9h07t7I_f_WqBzK5qblysvkkp7wNHiBVeJ5f37MdFUd6XQVWPilIrzXemqwMm9f3EbDvW3MOJDjOt7AJCKU1oTuQH8gjOv4jzedU4u8U2FtyfpLrVB6&partNumber=8"

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[###                           ] 12% data/file43
ERROR: failed to upload '.dvc/cache/e2/8b8c31cad20b27fd8d786c26019dfe' to 's3://<S3 bucket>/dvc_many_files_test/e2/8b8c31cad20b27fd8d786c26019dfe' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/e2/8b8c31cad20b27fd8d786c26019dfe'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[#                             ] 6% data/file26
ERROR: failed to upload '.dvc/cache/31/2236cd7f963add79a4a42082b804de' to 's3://<S3 bucket>/dvc_many_files_test/31/2236cd7f963add79a4a42082b804de' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/31/2236cd7f963add79a4a42082b804de'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] ?% data/file15
ERROR: failed to upload '.dvc/cache/26/dada6fb3c142dd8caf65143a701091' to 's3://<S3 bucket>/dvc_many_files_test/26/dada6fb3c142dd8caf65143a701091' - Could not connect to the endpoint URL: "https://<S3 bucket>.s3.eu-west-2.amazonaws.com/dvc_many_files_test/26/dada6fb3c142dd8caf65143a701091?uploadId=Pr0PDy3MYAwcdK61oUGyR.yl8qKT7OSmOsnu.yGz6lCPU4vS3YZGqd0Sj6iII9aON8En2N7ACtO9OvSCGdkbVs_j1JJBGbIEVDzzy0bJrO1fzAo1juqJTMqv9FzTf3Sm&partNumber=10"

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!

[                              ] ?% data/file29ERROR: failed to upload '.dvc/cache/3c/bdb8ba8d22523ff1f9d80438fe805d' to 's3://<S3 bucket>/dvc_many_files_test/3c/bdb8ba8d22523ff1f9d80438fe805d' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/3c/bdb8ba8d22523ff1f9d80438fe805d'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] ?% data/file28
ERROR: failed to upload '.dvc/cache/64/1af74eb5d1c157c4ebd47423655c6e' to 's3://<S3 bucket>/dvc_many_files_test/64/1af74eb5d1c157c4ebd47423655c6e' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/64/1af74eb5d1c157c4ebd47423655c6e'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] 0% data/file9
ERROR: failed to upload '.dvc/cache/de/ec9b8e358242912c7fcb516146c764' to 's3://<S3 bucket>/dvc_many_files_test/de/ec9b8e358242912c7fcb516146c764' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/de/ec9b8e358242912c7fcb516146c764'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[                              ] ?% data/file98
ERROR: failed to upload '.dvc/cache/4c/055a0bac15aa0b28d6e8ff5b9bc537' to 's3://<S3 bucket>/dvc_many_files_test/4c/055a0bac15aa0b28d6e8ff5b9bc537' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/4c/055a0bac15aa0b28d6e8ff5b9bc537'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!

ERROR: failed to upload '.dvc/cache/c0/e9fd46cca1f0fe6dfd759196fc6fc9' to 's3://<S3 bucket>/dvc_many_files_test/c0/e9fd46cca1f0fe6dfd759196fc6fc9' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/c0/e9fd46cca1f0fe6dfd759196fc6fc9'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[###                           ] 10% data/file90
ERROR: failed to upload '.dvc/cache/83/e180e32a4a4d9664f5c2464fd0cf35' to 's3://<S3 bucket>/dvc_many_files_test/83/e180e32a4a4d9664f5c2464fd0cf35' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/83/e180e32a4a4d9664f5c2464fd0cf35'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[####                          ] 14% data/file91
ERROR: failed to upload '.dvc/cache/6e/c586afabe4e13f1adb12f168d7179c' to 's3://<S3 bucket>/dvc_many_files_test/6e/c586afabe4e13f1adb12f168d7179c' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/6e/c586afabe4e13f1adb12f168d7179c'

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[#                             ] 4% data/file9
ERROR: failed to upload '.dvc/cache/16/c254f07f818e0dc9c552729d436ba4' to 's3://<S3 bucket>/dvc_many_files_test/16/c254f07f818e0dc9c552729d436ba4' - [Errno 24] Too many open files: '/<user>/dvc_many_files_test/.dvc/cache/16/c254f07f818e0dc9c552729d436ba4'

[...]

~/dvc_many_files_test $
```
Some observations:
* When trying to reproduce this, it seems significant that the files are of a decent size (in this case 100MB)
* Reproducing by creating files from `/dev/zero` instead of `/dev/urandom` doesn't seem to work (I think there is some background compression or trick for this low entropy file)
* Some of the logs indicate errors uploading a part of the file (search `partNumber`), suggesting multi-threaded upload.
@ChrisHowlin Sorry for the late response. Thank you for the investigation! :slightly_smiling_face:  So to summarize, while we are taking a closer look at this, a workaround for you would be to either increase ulimit or lower `--jobs` on `push/pull`. 

For the record: I am able to reproduce with your script on my mac.
I can confirm the above workarounds. To summarise (for anyone else experiencing this), I was able to work around the issue with either of these commands:
* `dvc push --jobs 8` - Reducing number of jobs to 2*CPUs (8 in my case)
* `ulimit -n 4096` - Increasing OS limit on open files per proc (run `dvc push` after setting this)
@efiop , are we leaking some file descriptors? is this expected in macos? (we don't have anything related to `ulimit` in our docs)

Commit message: remote: s3: revert jobs adjustment, pass OSError to main

Patch to review:
diff --git a/dvc/main.py b/dvc/main.py
index 86e1a9e0df..0b4f95344a 100644
--- a/dvc/main.py
+++ b/dvc/main.py
@@ -1,6 +1,7 @@
 """Main entry point for dvc CLI."""
 from __future__ import unicode_literals
 
+import errno
 import logging
 
 from dvc import analytics
@@ -64,6 +65,10 @@ def main(argv=None):
                 "unicode is not supported in DVC for Python 2 "
                 "(end-of-life January 1, 2020), please upgrade to Python 3"
             )
+        elif isinstance(exc, OSError) and exc.errno == errno.EMFILE:
+            logger.exception(
+                "too many open files error, please increase you `ulimit`"
+            )
         else:
             logger.exception("unexpected error")
         ret = 255
diff --git a/dvc/remote/base.py b/dvc/remote/base.py
index f72e56127d..57a1846a66 100644
--- a/dvc/remote/base.py
+++ b/dvc/remote/base.py
@@ -1,4 +1,7 @@
 from __future__ import unicode_literals
+
+import errno
+
 from dvc.utils.compat import basestring, FileNotFoundError, str, urlparse
 
 import itertools
@@ -516,6 +519,16 @@ def _save(self, path_info, checksum):
             return
         self._save_file(path_info, checksum)
 
+    def _handle_transfer_exception(
+        self, from_info, to_info, exception, operation
+    ):
+        if isinstance(exception, OSError) and exception.errno == errno.EMFILE:
+            raise exception
+
+        msg = "failed to {} '{}' to '{}'".format(operation, from_info, to_info)
+        logger.exception(msg)
+        return 1
+
     def upload(self, from_info, to_info, name=None, no_progress_bar=False):
         if not hasattr(self, "_upload"):
             raise RemoteActionNotImplemented("upload", self.scheme)
@@ -537,10 +550,10 @@ def upload(self, from_info, to_info, name=None, no_progress_bar=False):
                 name=name,
                 no_progress_bar=no_progress_bar,
             )
-        except Exception:
-            msg = "failed to upload '{}' to '{}'"
-            logger.exception(msg.format(from_info, to_info))
-            return 1  # 1 fail
+        except Exception as e:
+            return self._handle_transfer_exception(
+                from_info, to_info, e, "upload"
+            )
 
         return 0
 
@@ -614,10 +627,10 @@ def _download_file(
             self._download(
                 from_info, tmp_file, name=name, no_progress_bar=no_progress_bar
             )
-        except Exception:
-            msg = "failed to download '{}' to '{}'"
-            logger.exception(msg.format(from_info, to_info))
-            return 1  # 1 fail
+        except Exception as e:
+            return self._handle_transfer_exception(
+                from_info, to_info, e, "download"
+            )
 
         move(tmp_file, to_info, mode=file_mode)
 
diff --git a/tests/func/test_remote.py b/tests/func/test_remote.py
index 23f318861b..f3140b5f33 100644
--- a/tests/func/test_remote.py
+++ b/tests/func/test_remote.py
@@ -1,14 +1,17 @@
+import errno
 import os
 import shutil
 
 import configobj
+import pytest
 from mock import patch
 
 from dvc.config import Config
 from dvc.main import main
 from dvc.path_info import PathInfo
-from dvc.remote import RemoteLOCAL
+from dvc.remote import RemoteLOCAL, RemoteConfig
 from dvc.remote.base import RemoteBASE
+from dvc.utils.compat import fspath
 from tests.basic_env import TestDvc
 from tests.remotes import get_local_url, get_local_storagepath
 
@@ -253,3 +256,21 @@ def unreliable_upload(self, from_file, to_info, name=None, **kwargs):
 def get_last_exc(caplog):
     _, exc, _ = caplog.records[-2].exc_info
     return exc
+
+
+def test_raise_on_too_many_open_files(tmp_dir, dvc, tmp_path_factory, mocker):
+    storage = tmp_path_factory.mktemp("test_remote_base")
+    remote_config = RemoteConfig(dvc.config)
+    remote_config.add("local_remote", fspath(storage), default=True)
+
+    tmp_dir.dvc_gen({"file": "file content"})
+
+    mocker.patch.object(
+        RemoteLOCAL,
+        "_upload",
+        side_effect=OSError(errno.EMFILE, "Too many open files"),
+    )
+
+    with pytest.raises(OSError) as e:
+        dvc.push()
+        assert e.errno == errno.EMFILE


Code review comments:
diff_hunk: @@ -64,6 +65,10 @@ def main(argv=None):
                 "unicode is not supported in DVC for Python 2 "
                 "(end-of-life January 1, 2020), please upgrade to Python 3"
             )
+        elif isinstance(exc, OSError) and exc.errno == errno.EMFILE:
+            logger.exception(
+                "too many open files error, please increase you `ulimit`"
path: dvc/main.py
text: @user1:
```suggestion
                "too many open files, please increase your `ulimit`"
```