preprocessing too memory heavy and complex. Modularizing into separate files