AstroGPU/swarm — Issue #14: If swarm creates a large log file, then it can die at the end when it tries to create the sorted output and/or indices
Author: eford
Created: 2012-03-01 15:37:50+00:00
State: open

BTW- I found the place where the code uses a ton of memory.
It's after the integration is done and it sort's the raw binary file
to make a collated binary file.
I think the problem is that if you have written a lot of data to disk
(say ~30GB) in your .bin.raw file, then when it tries to sort the file
it runs out of RAM.
Do you understand what's going on in swarm/log/io.cpp well enough to fix that?
I guess there are actually two things to do:
1) use a different sort algorithm, so you don't try to load the entire
data file into RAM at once, and
2) perhaps you could make it so that instead of automatically
converting the .bin.raw file into bin, there was a separate "swarm
collate file.bin.raw file.bin" command the takes a .bin.raw file and
outputs a collated .bin file (and maybe also generates the indexces).
That way if it did when it tried to sort, you could at least recover
your data

In src/swarm/log/io.cpp
template index_creator::finish
template index_creator::index_binary_log_file
template index_creator:: sort_binary_log_file

@salehqt