Switching over to manual streaming of Parquet output to local disk. We may want to put on our engineering hats and engineer a big fancy Spark Streaming solution, but this should work.