tap-git/tap — Issue #2: Preposterous startup time
Author: psutter
Created: 2011-12-15 19:00:27+00:00
State: open

When running a job with 60,000 input files stored in S3, it can take two hours to check for file existence in generatePlan before the job even beings.

Solutions:
1. Implement this as many threads, perhaps max(50, number of files / 30)
2. Have a flag to bypass this check
3. Automatically skip this check if there are >30 files, since this is almost certainly a list of files generated elsewhere.

private PipePlan generatePlan(List errors) {
...
while (!toGenerate.isEmpty()) {
Pipe file = toGenerate.iterator().next();
toGenerate.remove(file);
boolean exists = file.exists(baseConf);
if (exists && !file.isObsolete(baseConf)
&& (!forceRebuild || file.getProducer() == null)) {
if (!generated.contains(file)) {
System.out.println("File: " + file.getPath()
+ " exists and is up to date.");
// ok already
generated.add(file);
plan.fileCreateWith(file, null);
}