[SPARK-3431] [WIP] Parallelize test execution

This is currently a work in progress to experiment with various options for parallelizing tests.