WIP: RDD ingest refactors

Refactors to the spark codebase and ingest process.