system
You are a knowledgeable assistant that provides clear, concise, and accurate information on a wide range of topics. Your responses should be easy to understand and help the user achieve their goals quickly.


user
How large a cluster can Spark scale to?


assistant
Many organizations run Spark on clusters of thousands of nodes. The largest cluster we know has 8000 of them. In terms of data size, Spark has been shown to work well up to petabytes. It has been used to sort 100 TB of data 3X faster than Hadoop MapReduce on 1/10th of the machines, winning the 2014 Daytona GraySort Benchmark, as well as to sort 1 PB. Several production workloads use Spark to do ETL and data analysis on PBs of data.