git/git — pull request #115 by KingDuckZ (2014-11-11 20:23:27+00:00)
Title: Import files in chunks to prevent p4 from opening too many files.

While doing a git p4 sync, I ran into a very large commit adding boost headers to the repository. In spite of me increasing the maximum open files limit, I kept on getting errors. Note that in my test I added the "Max files I can open" message to print some useful values. This output demonstrates the problem:

$ git p4 sync --max-changes 2
Performing incremental import into refs/remotes/p4/master git branch
Depot paths: //depot/
Import destination: refs/remotes/p4/master
Importing revision 42494 (50%)Max files I can open is 16384, readAtTime set to 1381, trying to read 1381 files
Perforce client error:
Partner exited unexpectedly.
Error from p4 print for //depot/extra/boost/preprocessor/seq/subseq.hpp: Date 2014/11/07 12:31:43:
Operation: flush2
Operation 'open for read' failed.
Database open error on pdb.lbr!
open for read: depot/depot/extra/boost/preprocessor/seq/subseq.hpp,v: Too many open files
open for read: depot/depot/extra/boost/preprocessor/seq/subseq.hpp,v: Too many open files

$ ulimit -Hn
32768
$ ulimit -Sn
16384

My understanding is that p4 tries to open all of the files it receives on the command line and thus hits the system limit. Running git-p4 with my fix solved this issue for me, I could get past the large commit and complete the sync without touching at the file limit.
I'm not very familiar with git-p4, so I hope the fix is correct in all circumstances! Also, I tried to dynamically obtain the max number of open files from the system, but I kept on getting the error and ended up hardcoding 256 as the limit. Most commits should include way fewer files than that anyways, so performance should still be more or less the same.