What is Apache Spark?PlatformThe Databricks Lakehouse PlatformDelta LakeData GovernanceData EngineeringData StreamingData WarehousingData SharingMachine LearningData SciencePricingMarketplaceOpen source techSecurity and Trust CenterWEBINAR May 18 / 8 AM PT
Goodbye, Data Warehouse. Hello, Lakehouse.
Attend to understand how a data lakehouse fits within your modern data stack.
Register nowSolutionsSolutions by IndustryFinancial ServicesHealthcare and Life SciencesManufacturingCommunications, Media & EntertainmentPublic SectorRetailSee all IndustriesSolutions by Use CaseSolution AcceleratorsProfessional ServicesDigital Native BusinessesData Platform MigrationNew survey of biopharma executives reveals real-world success with real-world evidence.
See survey resultsLearnDocumentationTraining & CertificationDemosResourcesOnline CommunityUniversity AllianceEventsData + AI SummitBlogLabsBeaconsJoin Generation AI in San Francisco
June 26–29
 
Learn about LLMs like Dolly and open source Data and AI technologies such as Apache Spark™, Delta Lake, MLflow and Delta SharingExplore sessionsCustomersPartnersCloud PartnersAWSAzureGoogle CloudPartner ConnectTechnology and Data PartnersTechnology Partner ProgramData Partner ProgramBuilt on Databricks Partner ProgramConsulting & SI PartnersC&SI Partner ProgramPartner SolutionsConnect with validated partner solutions in just a few clicks.Learn moreCompanyCareers at DatabricksOur TeamBoard of DirectorsCompany BlogNewsroomDatabricks VenturesAwards and RecognitionContact UsSee why Gartner named Databricks a Leader for the second consecutive yearGet the reportTry DatabricksWatch DemosContact UsLoginJUNE 26-29REGISTER NOWApache SparkAll>Apache SparkTry Databricks for freeGet StartedWhat Is Apache Spark?Apache Spark is an open source analytics engine used for big data workloads. It can handle both batches as well as real-time analytics and data processing workloads. Apache Spark started in 2009 as a research project at the University of California, Berkeley. Researchers were looking for a way to speed up processing jobs in Hadoop systems. It is based on Hadoop MapReduce and it extends the MapReduce model to efficiently use it for more types of computations, which includes interactive queries and stream processing. Spark provides native bindings for the Java, Scala, Python, and R programming languages. In addition, it includes several libraries to support build applications for machine learning [MLlib], stream processing [Spark Streaming], and graph processing [GraphX]. Apache Spark consists of Spark Core and a set of libraries. Spark Core is the heart of Apache Spark and it is responsible for providing distributed task transmission, scheduling, and I/O functionality. The Spark Core engine uses the concept of a Resilient Distributed Dataset (RDD) as its basic data type. The RDD is designed so it will hide most of the computational complexity from its users. Spark is intelligent on the way it operates on data; data and partitions are aggregated across a server cluster, where it can then be computed and either moved to a different data store or run through an analytic model. You will not be asked to specify the destination of the files or the computational resources that need to be used in order to store or retrieve files. What Are the Benefits of Apache Spark?SpeedSpark executes very fast by caching data in memory across multiple parallel operations. The main feature of Spark is its in-memory engine that increases the processing speed; making it up to 100 times faster than MapReduce when processed in-memory, and 10 times faster on disk,  when it comes to large scale data processing. Spark makes this possible by reducing the number of reading/writing to disk operations.Real-time stream processingApache Spark can handle real-time streaming along with the integration of other frameworks. Spark ingests data in mini-batches and performs RDD transformations on those mini-batches of data.Supports Multiple WorkloadsApache Spark can run multiple workloads, including interactive queries, real-time analytics, machine learning, and graph processing. One application can combine multiple workloads seamlessly.Increased UsabilityThe ability to support several programming languages makes it dynamic. It allows you to quickly write applications in Java, Scala, Python, and R; giving you a variety of languages for building your applications.Advanced AnalyticsSpark supports SQL queries, machine learning, stream processing, and graph processing.Additional ResourcesAbout Apache SparkLearning Apache Spark 2nd Edition eBook8 Steps for a Developer to Learn Apache Spark with Delta Lake eBookBack to GlossaryProductPlatform OverviewPricingOpen Source TechTry DatabricksDemoProductPlatform OverviewPricingOpen Source TechTry DatabricksDemoLearn & SupportDocumentationGlossaryTraining & CertificationHelp CenterLegalOnline CommunityLearn & SupportDocumentationGlossaryTraining & CertificationHelp CenterLegalOnline CommunitySolutionsBy IndustriesProfessional ServicesSolutionsBy IndustriesProfessional ServicesCompanyAbout UsCareers at DatabricksDiversity and InclusionCompany BlogContact UsCompanyAbout UsCareers at DatabricksDiversity and InclusionCompany BlogContact UsSee Careers
at DatabricksWorldwideEnglish (United States)Deutsch (Germany)Français (France)Italiano (Italy)日本語 (Japan)한국어 (South Korea)Português (Brazil)Databricks Inc.
160 Spear Street, 13th Floor
San Francisco, CA 94105
1-866-330-0121© Databricks 2023. All rights reserved. Apache, Apache Spark, Spark and the Spark logo are trademarks of the Apache Software Foundation.Privacy Notice|Terms of Use|Your Privacy Choices|Your California Privacy Rights