code cleanup and modularization, transition to bulk processing