So it turns out that positive feedback loops consume 600GB of space, fill the partition, and then die. Oops.
final fixes to the awk strings. Now at the stage where the only stupid in the data is actually in the data.