refactored preprocessing into single pass and minor improvements