big refactor to main method signatures, added more tests, mass stream stress / data integrity test still fails with big rows