Materialized viewsPlatformThe Databricks Lakehouse PlatformDelta LakeData GovernanceData EngineeringData StreamingData WarehousingData SharingMachine LearningData SciencePricingMarketplaceOpen source techSecurity and Trust CenterWEBINAR May 18 / 8 AM PT
Goodbye, Data Warehouse. Hello, Lakehouse.
Attend to understand how a data lakehouse fits within your modern data stack.
Register nowSolutionsSolutions by IndustryFinancial ServicesHealthcare and Life SciencesManufacturingCommunications, Media & EntertainmentPublic SectorRetailSee all IndustriesSolutions by Use CaseSolution AcceleratorsProfessional ServicesDigital Native BusinessesData Platform MigrationNew survey of biopharma executives reveals real-world success with real-world evidence.
See survey resultsLearnDocumentationTraining & CertificationDemosResourcesOnline CommunityUniversity AllianceEventsData + AI SummitBlogLabsBeaconsJoin Generation AI in San Francisco
June 26–29
 
Learn about LLMs like Dolly and open source Data and AI technologies such as Apache Spark™, Delta Lake, MLflow and Delta SharingExplore sessionsCustomersPartnersCloud PartnersAWSAzureGoogle CloudPartner ConnectTechnology and Data PartnersTechnology Partner ProgramData Partner ProgramBuilt on Databricks Partner ProgramConsulting & SI PartnersC&SI Partner ProgramPartner SolutionsConnect with validated partner solutions in just a few clicks.Learn moreCompanyCareers at DatabricksOur TeamBoard of DirectorsCompany BlogNewsroomDatabricks VenturesAwards and RecognitionContact UsSee why Gartner named Databricks a Leader for the second consecutive yearGet the reportTry DatabricksWatch DemosContact UsLoginJUNE 26-29REGISTER NOWMaterialized viewsAll>Materialized viewsTry Databricks for freeGet StartedDelta Pipelines / Materialized Views in Databricks DeltaIntroDelta Pipelines provides a set of APIs and UI for managing the data pipeline lifecycle. This open-source framework helps data engineering teams simplify ETL development, improve data reliability, and scale operations. Instead of coding transformations and scheduling jobs for your data, you can easily define the end state you want for your data by building declarative pipelines.  You can chain the dependencies between various tasks such that updating one table will automatically trigger all downstream reprocessing. With Delta Lake, you will have reliable data lakes; with delta pipelines, you will have reliable data pipelines for your data workflows.Delta Pipelines DatasetsThe primary abstraction in Delta Pipelines is what’s known as a Dataset. A Dataset is basically a query or view of an underlying table that can be expressed using APIs.  Below is an example using the Scala API:
dataset("events")
   .query {
       input("base_table")
       .select(...)
       .filter(...)
   }
Creating Materialized Views of DatasetsDelta Pipelines also offers the ability to create a materialized view of the data by simply chaining the .materialize() operator on to the dataset set like such:
dataset("events")
  .query {
     input("base_table")
     .select(...)
     .filter(...)
  }
     .materialize()

This means that the “events” materialized view will be continuously updated as data flows through the Delta Pipeline. As new data arrives and is committed to the base table via ACID transactions, the materialized view is continuously updated incrementally for each of those transactions as well. Users need not worry about when or how frequently to refresh the materialized view or even things like when to run OPTIMIZE or VACUUM commands -- it’s all done in the background automatically.See the below video for more details on Delta PipelinesTo play this video, click here and accept cookiesBack to GlossaryProductPlatform OverviewPricingOpen Source TechTry DatabricksDemoProductPlatform OverviewPricingOpen Source TechTry DatabricksDemoLearn & SupportDocumentationGlossaryTraining & CertificationHelp CenterLegalOnline CommunityLearn & SupportDocumentationGlossaryTraining & CertificationHelp CenterLegalOnline CommunitySolutionsBy IndustriesProfessional ServicesSolutionsBy IndustriesProfessional ServicesCompanyAbout UsCareers at DatabricksDiversity and InclusionCompany BlogContact UsCompanyAbout UsCareers at DatabricksDiversity and InclusionCompany BlogContact UsSee Careers
at DatabricksWorldwideEnglish (United States)Deutsch (Germany)Français (France)Italiano (Italy)日本語 (Japan)한국어 (South Korea)Português (Brazil)Databricks Inc.
160 Spear Street, 13th Floor
San Francisco, CA 94105
1-866-330-0121© Databricks 2023. All rights reserved. Apache, Apache Spark, Spark and the Spark logo are trademarks of the Apache Software Foundation.Privacy Notice|Terms of Use|Your Privacy Choices|Your California Privacy Rights