hoxu/gitstats — pull request #27 by amotl (2013-11-03 01:35:12+00:00)
Title: fix for processing multiple large repositories: let process pool for line count data collection honor the "processes" configuration setting

Currently, gitstats croaks at various places when running across multiple large repositories at once, on Linux as well as on OSX (see below). While raising ulimit values seemed to help at first, we found honoring the process pool size from the configuration settings would actually be a better solution, since we still seemed to run into oom situations. Do you agree?

Cheers and thanks for this great tool,
-- Andreas

```
Linux 1:
File "../parts/gitstats/gitstats", line 281, in collect
self.total_authors += int(getpipeoutput(['git shortlog -s %s' % getcommitrange(), 'wc -l']))
File "../parts/gitstats/gitstats", line 62, in getpipeoutput
p = subprocess.Popen(x, stdin = p0.stdout, stdout = subprocess.PIPE, shell = True)
[...]
File "/usr/lib64/python2.7/subprocess.py", line 1143, in _execute_child
self.pid = os.fork()
OSError: [Errno 11] Resource temporarily unavailable

Linux 2:
File "../parts/gitstats/gitstats", line 452, in collect
time_rev_count = Pool(processes=conf['processes']).map(getnumoffilesfromrev, revs_to_read)
[...]
File "/usr/lib64/python2.7/threading.py", line 494, in start
_start_new_thread(self.__bootstrap, ())
thread.error: can't start new thread

OSX:
File "../parts/gitstats/gitstats", line 510, in collect
ext_blob_linecount = Pool(processes=24).map(getnumoflinesinblob, blobs_to_read)
[...]
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/forking.py", line 121, in __init__
self.pid = os.fork()
OSError: [Errno 35] Resource temporarily unavailable
```