Announcing General Availability of Data lineage in Unity Catalog - The Databricks BlogSkip to main contentPlatformThe Databricks Lakehouse PlatformDelta LakeData GovernanceData EngineeringData StreamingData WarehousingData SharingMachine LearningData SciencePricingMarketplaceOpen source techSecurity and Trust CenterWEBINAR May 18 / 8 AM PT
Goodbye, Data Warehouse. Hello, Lakehouse.
Attend to understand how a data lakehouse fits within your modern data stack.
Register nowSolutionsSolutions by IndustryFinancial ServicesHealthcare and Life SciencesManufacturingCommunications, Media & EntertainmentPublic SectorRetailSee all IndustriesSolutions by Use CaseSolution AcceleratorsProfessional ServicesDigital Native BusinessesData Platform MigrationNew survey of biopharma executives reveals real-world success with real-world evidence.
See survey resultsLearnDocumentationTraining & CertificationDemosResourcesOnline CommunityUniversity AllianceEventsData + AI SummitBlogLabsBeaconsJoin Generation AI in San Francisco
June 26–29
 
Learn about LLMs like Dolly and open source Data and AI technologies such as Apache Spark™, Delta Lake, MLflow and Delta SharingExplore sessionsCustomersPartnersCloud PartnersAWSAzureGoogle CloudPartner ConnectTechnology and Data PartnersTechnology Partner ProgramData Partner ProgramBuilt on Databricks Partner ProgramConsulting & SI PartnersC&SI Partner ProgramPartner SolutionsConnect with validated partner solutions in just a few clicks.Learn moreCompanyCareers at DatabricksOur TeamBoard of DirectorsCompany BlogNewsroomDatabricks VenturesAwards and RecognitionContact UsSee why Gartner named Databricks a Leader for the second consecutive yearGet the reportTry DatabricksWatch DemosContact UsLoginJUNE 26-29REGISTER NOWCategoriesAll blog postsCompanyCultureCustomersEventsNewsPlatformAnnouncementsPartnersProductSolutionsSecurity and TrustEngineeringData Science and MLOpen SourceSolutions AcceleratorsData EngineeringTutorialsData StreamingData WarehousingData StrategyBest PracticesData LeaderInsightsIndustriesFinancial ServicesHealth and Life SciencesMedia and EntertainmentRetailManufacturingPublic SectorAnnouncing General Availability of Data lineage in Unity Catalogby Tao Feng, Paul Roome, Sachin Thakur, Hongyi Zhang, Menglei Sun and Shuai LuDecember 12, 2022 in Platform BlogShare this postToday, we are excited to announce the general availability of data lineage in Unity Catalog, available on AWS and Azure. With data lineage general availability, you can expect the highest level of stability, support, and enterprise readiness from Databricks for mission-critical workloads on the Databricks Lakehouse Platform. Refer the data lineage guides (AWS | Azure) to get started.In this blog, we explore how organizations leverage data lineage as a key lever of a pragmatic data governance strategy, some of the key features available in the GA release, and how to get started with data lineage in Unity Catalog.Driving better data observability and compliance with data lineageUnity Catalog provides a unified governance solution for data, analytics and AI, empowering data teams to catalog all their data and AI assets, define fine-grained access permissions using a familiar interface based on ANSI SQL, audit data access and share data across clouds, regions and data platforms.With automated data lineage in Unity Catalog, data teams can now automatically track sensitive data for compliance requirements and audit reporting, ensure data quality across all workloads, perform impact analysis or change management of any data changes across the lakehouse and conduct root cause analysis of any errors in their data pipelines. Learn more about common use cases for data lineage in our previous blog."Data Lineage has enabled us to get insights into how our datasets are used and by whom. This serves as both basic documentation as well as identifies who would be affected by dataset changes or deprecations to cut down on incidents" — Sam Shuster, Staff Engineer, Edmunds"Lineage is the last crucial piece for access control. It allows analysts to leverage data to do their jobs while adhering to all usage standards and access controls, even when recreating tables and data sets in another environment" — Chris Locklin, Data Platform Manager, Grammarly“Lineage helps Milliman professionals see where data is coming from, what transformations did it go through and how it is being used for the life of the project. This well-documented end-to-end process complements the standard actuarial process” — Dan McCurley, Cloud Solutions Architect, MillimanKey features of data lineage available in the GA releaseAutomated real-time lineage: Unity Catalog automatically captures and displays data flow diagrams in real-time for queries executed in any language (Python, SQL, R, and Scala) and execution mode (batch and streaming). Real-time lineage reduces the operational overhead of manually creating data flow trails. Data lineage is automatically aggregated across all workspaces connected to a Unity Catalog metastore, this means that lineage captured in one workspace can be seen in any other workspace that shares the same metastore.Unified column and table lineage graph: With Unity Catalog, users can now see both column and table lineage in a single lineage graph, giving users a better understanding of what a particular table or column is made up of and where the data is coming from. Users can navigate the lineage graph upstream or downstream with a few clicks to see the full data flow diagram.Table and column lineage in Unity CatalogGoing beyond just tables and columns: Unity Catalog also tracks lineage for notebooks, workflows, and dashboards. This improves end-to-end visibility into how data is used in your organization and allows you to understand the impact of any data changes on downstream consumers.Lineage for notebooks, workflows and dashboardsBuilt-in security: Lineage graphs are secure by default and use the Unity Catalog's common permission model. Users must have the appropriate permissions to view the lineage data flow diagram, adding an extra layer of security and reducing the risk of unintentional data breaches. For example, if users do not have the SELECT privilege on a table, they will be unable to explore the table's lineage. Similarly, users can only see lineage information for notebooks, workflows, and dashboards that they have permission to view.Built-in security for lineage graphsPartner integrations: Unity Catalog also offers rich integration with various data governance partners via Unity Catalog REST APIs, enabling easy export of lineage information.Getting started with data lineage in Unity CatalogWatch the demo below to see data lineage in action.To play this video, click here and accept cookiesData lineage is included at no extra cost with Databricks Premium and Enterprise tiers. All workloads referencing the Unity Catalog metastore now have data lineage enabled by default, and all workloads reading or writing to Unity Catalog will automatically capture lineage. To take advantage of automatically captured Data Lineage, please restart any clusters or SQL Warehouses that were started prior to December 7th, 2022. If you already have a Databricks account, you can get started by following the data lineage guides (AWS | Azure). If you are not an existing Databricks customer, sign up for a free trial with a Premium or Enterprise workspace.Try Databricks for freeGet StartedRelated postsAnnouncing the Availability of Data Lineage With Unity CatalogJune 8, 2022 by Paul Roome, Tao Feng and Sachin Thakur in Platform Blog
Update: Data Lineage is now generally available on AWS and Azure. We are excited to announce that data lineage for Unity Catalog, the...
Simplify Access Policy Management With Privilege Inheritance in Unity CatalogOctober 21, 2022 by Jim Thorstad, Paul Roome and Sachin Thakur in Platform Blog
Unity Catalog, now generally available on AWS and Azure, provides a unified governance solution for data, analytics and AI on the lakehouse. Thousands...
Announcing General Availability of Delta SharingAugust 26, 2022 by Matei Zaharia, Celia Kung, Xiaotong Sun, Steve Mahoney, Itai Weiss, Sachin Thakur and Jay Bhankharia in Announcements
Today we are excited to announce that Delta Sharing is generally available (GA) on AWS and Azure. With the GA release, you can...
See all Platform Blog postsProductPlatform OverviewPricingOpen Source TechTry DatabricksDemoProductPlatform OverviewPricingOpen Source TechTry DatabricksDemoLearn & SupportDocumentationGlossaryTraining & CertificationHelp CenterLegalOnline CommunityLearn & SupportDocumentationGlossaryTraining & CertificationHelp CenterLegalOnline CommunitySolutionsBy IndustriesProfessional ServicesSolutionsBy IndustriesProfessional ServicesCompanyAbout UsCareers at DatabricksDiversity and InclusionCompany BlogContact UsCompanyAbout UsCareers at DatabricksDiversity and InclusionCompany BlogContact UsSee Careers
at DatabricksWorldwideEnglish (United States)Deutsch (Germany)Français (France)Italiano (Italy)日本語 (Japan)한국어 (South Korea)Português (Brazil)Databricks Inc.
160 Spear Street, 13th Floor
San Francisco, CA 94105
1-866-330-0121© Databricks 2023. All rights reserved. Apache, Apache Spark, Spark and the Spark logo are trademarks of the Apache Software Foundation.Privacy Notice|Terms of Use|Your Privacy Choices|Your California Privacy Rights