What is a Spark API?PlatformThe Databricks Lakehouse PlatformDelta LakeData GovernanceData EngineeringData StreamingData WarehousingData SharingMachine LearningData SciencePricingMarketplaceOpen source techSecurity and Trust CenterWEBINAR May 18 / 8 AM PT
Goodbye, Data Warehouse. Hello, Lakehouse.
Attend to understand how a data lakehouse fits within your modern data stack.
Register nowSolutionsSolutions by IndustryFinancial ServicesHealthcare and Life SciencesManufacturingCommunications, Media & EntertainmentPublic SectorRetailSee all IndustriesSolutions by Use CaseSolution AcceleratorsProfessional ServicesDigital Native BusinessesData Platform MigrationNew survey of biopharma executives reveals real-world success with real-world evidence.
See survey resultsLearnDocumentationTraining & CertificationDemosResourcesOnline CommunityUniversity AllianceEventsData + AI SummitBlogLabsBeaconsJoin Generation AI in San Francisco
June 26–29
 
Learn about LLMs like Dolly and open source Data and AI technologies such as Apache Spark™, Delta Lake, MLflow and Delta SharingExplore sessionsCustomersPartnersCloud PartnersAWSAzureGoogle CloudPartner ConnectTechnology and Data PartnersTechnology Partner ProgramData Partner ProgramBuilt on Databricks Partner ProgramConsulting & SI PartnersC&SI Partner ProgramPartner SolutionsConnect with validated partner solutions in just a few clicks.Learn moreCompanyCareers at DatabricksOur TeamBoard of DirectorsCompany BlogNewsroomDatabricks VenturesAwards and RecognitionContact UsSee why Gartner named Databricks a Leader for the second consecutive yearGet the reportTry DatabricksWatch DemosContact UsLoginJUNE 26-29REGISTER NOWSpark APIAll>Spark APITry Databricks for freeGet StartedIf you are working with Spark, you will come across the three APIs: DataFrames, Datasets, and RDDsWhat are Resilient Distributed Datasets?RDD or Resilient Distributed Datasets, is a collection of records with distributed computing, which are fault tolerant, immutable in nature. They can be operated on in parallel with low-level APIs, while their lazy feature makes the spark operation to work at an improved speed. RDDs support two types of operations:Transformations - lazy operations that return another RDD, this RDD doesn’t compute unless action is performed on it. Some examples of transformations are map(), flatmap(), filter() Actions - operations that trigger computation and return values. Some examples of actions are count, top(), savetofile() Disadvantages of RDDsIf you choose to work with RDD you will have to optimize each and every RDD. In addition, unlike Datasets and DataFrames, RDDs don’t infer the schema of the data ingested therefore you will have to specify it.What are DataFrames?DataFrames is a distributed collection of rows under named columns. In simple terms, it looks like an Excel sheet with Column headers, or you can think of it as the equivalent to a table in a relational database or a DataFrame in R or Python. It has three main common characteristics with RDD:Immutable in nature: You will be able to create a DataFrame but you will not be able to change it. A DataFrame just like an RDD can be transformedLazy Evaluations:  a task is not executed until an action is performed.Distributed:  DataFrames just like RDDs are both distributed in nature.Ways to Create a DataFrameIn Spark DataFrames can be created in several ways:Using different data formats. Such as loading the data from JSON, CSV, RDBMS, XML or ParquetLoading the data from an already existing RDD.Programmatically specifying schemaDisadvantages of DataFramesThe main drawback of DataFrame API is that it does not support compile time safely, as a result, the user is limited in case the structure of the data is not known.What are Datasets?A Dataset is a strongly-typed, immutable collection of objects that are mapped to a relational schema. A Dataset can be created using JVM objects and manipulated using complex functional transformations. Datasets can be created in two ways:DynamicallyReading from a JSON file using SparkSession.Disadvantages of DataSetsThe main disadvantage of datasets is that they require typecasting into strings.Additional Resources8 Steps for a Developer to Learn Apache Spark with Delta Lake eBookThe Data Engineer's Guide to Apache Spark and Delta Lake eBookComparing Apache Spark and DatabricksBack to GlossaryProductPlatform OverviewPricingOpen Source TechTry DatabricksDemoProductPlatform OverviewPricingOpen Source TechTry DatabricksDemoLearn & SupportDocumentationGlossaryTraining & CertificationHelp CenterLegalOnline CommunityLearn & SupportDocumentationGlossaryTraining & CertificationHelp CenterLegalOnline CommunitySolutionsBy IndustriesProfessional ServicesSolutionsBy IndustriesProfessional ServicesCompanyAbout UsCareers at DatabricksDiversity and InclusionCompany BlogContact UsCompanyAbout UsCareers at DatabricksDiversity and InclusionCompany BlogContact UsSee Careers
at DatabricksWorldwideEnglish (United States)Deutsch (Germany)Français (France)Italiano (Italy)日本語 (Japan)한국어 (South Korea)Português (Brazil)Databricks Inc.
160 Spear Street, 13th Floor
San Francisco, CA 94105
1-866-330-0121© Databricks 2023. All rights reserved. Apache, Apache Spark, Spark and the Spark logo are trademarks of the Apache Software Foundation.Privacy Notice|Terms of Use|Your Privacy Choices|Your California Privacy Rights