Load into Cassandra or BerkeleyJE. Fetch files, load, repeat.