cvmfs/cvmfs — pull request #567 by reneme (2014-10-16 13:24:52+00:00)
Title: FIX: Unbound Upload Scheduling Exhausts File Descriptors

The scheduling of uploads didn't have a flow control and could therefore exhaust the number of available file descriptors. This particularly happened when the uploading didn't keep up with the processing speed and the number of files to be published was large. Now we throttle the file processing as soon as too many unfinished uploads are accumulating.

Unfortunately the actual number of required file descriptors is very hard to predict as this depends on the implementation on the concrete `Uploader` class, but at least now it has an upper bound that should be in a reasonable range. For 64 threads, one would expect to need around 10'000 file descriptors as a rough maximum.

**Note:** The actual fix is rather small (see changes in `upload_facility.cc`) but required some minor refactoring in order to make the threshold of *unfinished uploads* dependent on the number of CPU cores.