allow increasing the target manifest size via commandline.
note that bigger manifest on small files gives better compression.
add --max-manifest-size to the getopt call.
insert at most 500 files separately to avoid killing the node.
try to speedup the pseudopythonparser. Unsuccessfully.
Backed out changeset 4af5914452c8: did not provide better performance (it got worse).
note where we actually lose time (found by simply throwing in a chain of submethods calling and returning the next in the row, so the profiler could tell me the function).
refactor self.checkandprocesslinerest(unparsed) to self.checkandprocessunparsed() to simplify the model: always work on attributes, pass as little as possible.
pseudopythonparser: append to a self.unparsed list instead of appending to a string and join on demand. Speeds up self.readline() from 24s to 0.7s for huge freesites (2k files). Total runtime went down from >40s to 17s.