GitHub repository: virat02/Spark-Performance-Acceleration
Primary language: Scala
Created: 2019-12-12T00:44:51Z
Description: Implemented Riffle, an optimized shuffle service, efficiently merging fragmented intermediate shuffle files into larger block files. Developed NWayMerge, extending spark's internal code, converting small, random disk I/O requests into large, sequential ones. Achieved 50% reduction in disk reads and 10% improvement in the end-to-end job completion time.