JoeCarlson/intermine — pull request #3 by rnsmith (2014-10-22 15:18:26+00:00)
Title: Use a direct data loader for GATK VCF to avoid using items database.

This change uses a DirectDataLoader for the gatkvcf source to avoid using an items database and requirement for a post-process step.

A DirectDataLoader creates and stores InterMineObjects directly instead of first storing to an items database. This makes the data loading step much less efficient at merging objects. For every object that needs to be merged (has keys defined) a single query is executed in the target database, the standard way (with items) is to run these queries is large batches.

However, for this source the number of objects that need to be merged (genes, mRNAs, organism, etc) is tiny compared to the overall data size, so there is actually a big benefit in avoiding the items step.

With this method the Ptrichocarpa_210-initial16sample-GATK-filtered.vcf file loaded in just under two hours on one of our servers, including the SNPDiversitySamples (into a database that already contained genome annotation). Hopefully you see some improvement as well.

```
Total time: 116 minutes 40 seconds
org.intermine.dataloader.DirectDataLoader - Finished dataloading 113710347 objects at 978121 objects per minute (6975226 ms total) for source diversity-P.trichocarpa
```