What Is Data Governance? A Comprehensive Guide | DatabricksSkip to main contentPlatformThe Databricks Lakehouse PlatformDelta LakeData GovernanceData EngineeringData StreamingData WarehousingData SharingMachine LearningData SciencePricingMarketplaceOpen source techSecurity and Trust CenterWEBINAR May 18 / 8 AM PT
Goodbye, Data Warehouse. Hello, Lakehouse.
Attend to understand how a data lakehouse fits within your modern data stack.
Register nowSolutionsSolutions by IndustryFinancial ServicesHealthcare and Life SciencesManufacturingCommunications, Media & EntertainmentPublic SectorRetailSee all IndustriesSolutions by Use CaseSolution AcceleratorsProfessional ServicesDigital Native BusinessesData Platform MigrationNew survey of biopharma executives reveals real-world success with real-world evidence.
See survey resultsLearnDocumentationTraining & CertificationDemosResourcesOnline CommunityUniversity AllianceEventsData + AI SummitBlogLabsBeaconsJoin Generation AI in San Francisco
June 26–29
 
Learn about LLMs like Dolly and open source Data and AI technologies such as Apache Spark™, Delta Lake, MLflow and Delta SharingExplore sessionsCustomersPartnersCloud PartnersAWSAzureGoogle CloudPartner ConnectTechnology and Data PartnersTechnology Partner ProgramData Partner ProgramBuilt on Databricks Partner ProgramConsulting & SI PartnersC&SI Partner ProgramPartner SolutionsConnect with validated partner solutions in just a few clicks.Learn moreCompanyCareers at DatabricksOur TeamBoard of DirectorsCompany BlogNewsroomDatabricks VenturesAwards and RecognitionContact UsSee why Gartner named Databricks a Leader for the second consecutive yearGet the reportTry DatabricksWatch DemosContact UsLoginJUNE 26-29REGISTER NOWData GovernanceA comprehensive guide to the processes, policies and tech that organizations use to manage and get the most from their dataDownload free eBookIntroductionWhat is data governance?Data governance is a comprehensive approach that comprises the principles, practices and tools to manage an organization’s data assets throughout their lifecycle. By aligning data-related requirements with business strategy, data governance provides superior data management, quality, visibility, security and compliance capabilities across the organization. Implementing an effective data governance strategy allows companies to make data easily available for data-driven decision-making while safeguarding their data from unauthorized access, and ensuring compliance with regulatory requirements.What are the business benefits of data governance?Data governance is essential for unlocking the value of data, which is a critical asset for organizations. By implementing a robust data governance approach, businesses can leverage their data assets, gain a competitive edge, and earn and maintain customer trust by ensuring sound data and privacy practices.Increased operational efficiency and reduced costsEffective data governance allows organizations to create a single source of truth for their data estate, preventing data sprawl and silos, and reducing duplication. This leads to increased efficiency, reduced costs and easier management of security and governance concepts across the data estate.Improved productivity and faster decision-makingData governance promotes data democratization by ensuring data accuracy, consistency, and trustworthiness. It helps data users find high-quality data quickly, promoting a better understanding of the data’s meaning and context, leading to increased productivity and faster decision-making.Enhanced collaboration and value realizationA strong data governance program lays the foundation for enhanced data collaboration and sharing across teams, business units, and partners. This helps organizations promote knowledge sharing and build a better data culture, leading to increased innovation, better decision-making, and maximizing the value of their data.Enhanced security and privacyData governance mitigates security and privacy risks by implementing controls and processes to prevent unauthorized access and misuse of sensitive data. It promotes a culture of trust and transparency with stakeholders.Better compliance with regulations and standardsEffective data governance results in better compliance with regulatory requirements, such as HIPAA, FedRAMP, GDPR or CCPA. This protects the organization’s reputation, avoids potential financial and legal consequences, and increases stakeholder trust.Key elements of data governanceData catalogingEffective data governance requires knowledge of the data that exists within an organization. This is where a data catalog comes in, as it provides a centralized metadata repository for an organization’s data assets. A data catalog allows stakeholders to quickly discover, understand and access the data they need, improving data-related activities such as discovery, governance and analytics. It acts as a searchable index of all the data available, including information about its format, structure, location and usage, providing semantic value to an otherwise unidentifiable sea of information. Incorporating a data catalog into a governance program can help organizations improve their data management, enhance collaboration, reduce redundancy and ensure proper access controls and audit information retrieval. Data qualityIn today’s data-driven world, ensuring high data quality is crucial for accurate analytics, informed decision-making and cost-effectiveness. Data quality directly impacts the reliability of data-driven decisions and is a key aspect of data governance. To maintain effective data governance, organizations must prioritize the evaluation of key data quality attributes such as accuracy, completeness, freshness and compliance with data-quality rules. Therefore, a strong focus on data quality is essential in any data governance strategy, as it helps trace data lineage, enforce data quality rules, and track changes. Don’t let poor data quality compromise your business decisions and resource allocation — prioritize data quality as a critical part of your data governance efforts for better outcomes. Data classification Data classification is a crucial part of data governance that involves organizing and categorizing data based on its sensitivity, value and criticality. With the exponential growth of data, businesses are increasingly concerned about protecting sensitive data, mitigating risks and ensuring data quality. Classification allows organizations to identify and classify data based on its risk level and importance, allowing them to apply appropriate security measures and policies. A robust data classification system enhances data governance, reduces risks and ensures data quality and protection at scale. Data securityOrganizations understand the significance of granting high-quality data access to their teams to drive insights and business value, while prioritizing sensitive data protection against unauthorized access. Effective data access management is crucial for data security and governance, and a good data security governance program should include access controls that define which groups or individuals can access what data. These controls can be highly specific, down to the individual record or file. As data breaches and regulations such as GDPR and CCPA pose increased risks, businesses must establish clear governance policies that define who can access sensitive data sets and how to track any misuse. Unauthorized access to private or sensitive information should not occur, and implementing effective access management strategies is essential to safeguard data and maintain customer trust. Auditing data entitlements and accessEffective data access auditing is a critical aspect of data governance and security governance programs, particularly in regulated industries. By understanding who has access to what data and tracking recent access, organizations can proactively identify overentitled users or groups and adjust their access accordingly, minimizing the risk of data misuse. Without proper audit mechanisms in place, an organization may not be fully aware of their risk surface area, leaving them vulnerable to data breaches and regulatory noncompliance. Therefore, a well-designed audit team within a data governance or security governance organization plays a key role in ensuring data security and compliance with regulations such as GDPR and CCPA. By implementing effective data access auditing strategies, organizations can maintain the trust of their customers and protect their data from unauthorized access or misuse. Data lineageData lineage is a powerful tool that helps organizations ensure data quality and trustworthiness by providing a better understanding of data sources and consumption. It captures relevant metadata and events throughout the data’s lifecycle, providing an end-to-end view of how data flows across an organization’s data estate. As an essential pillar of a pragmatic data governance strategy, data lineage enables organizations to become compliant and audit-ready while reducing the operational overhead of creating audit trails manually, and providing trusted sources for audit reports. Additionally, data lineage empowers data consumers to perform better analyses, and helps data teams perform root cause analysis of any errors, significantly reducing debugging time. Data discoveryAs organizations continue to gather massive amounts of data from various sources, it’s becoming increasingly important to make this data easily discoverable for analytics, AI or ML use cases. This is critical to accelerate data democratization and unlock the true value of the data. Furthermore, with the emergence of modern data assets like dashboards, machine learning models, queries, libraries and notebooks, data discovery has become a key pillar of a robust data governance strategy. Organizations should view data discovery as a fundamental aspect of their data governance strategy. It enables data teams to easily locate data assets across the organization, collaborate on various projects, and innovate quickly and efficiently. This helps to prevent data duplication, which can be problematic as it costs money to persist them, and may lead to governance challenges at different security levels. Data sharing and collaboration Data sharing and collaboration are vital components in today’s business environment, with organizations exchanging data with internal teams, external partners, and customers across multiple clouds, data platforms and regions. As the demand for external data continues to grow, it is critical for organizations to securely exchange data while maintaining control and visibility over how their sensitive information is used. Data cleanrooms play a critical role in secure and controlled data collaboration, ensuring that data privacy regulations are upheld. It is essential for organizations to invest in open format, interoperable and multicloud data sharing technologies to meet their data-driven innovation needs. Moreover, data marketplaces serve as a bridge between data providers and consumers, facilitating the discovery and distribution of data sets. Therefore, it is crucial to recast data sharing as a business necessity and a crucial pillar of a robust data governance strategy.What does a good data governance solution look like?Data-forward organizations prioritize data, analytics and AI to drive business outcomes, and build their data strategies around a data lakehouse architecture, which unifies data, analytics and AI on a single platform. This architecture combines the best features of data warehouses and data lakes to handle all data, analytics and AI use cases. All data is stored in a cloud data lake and managed by a unified layer, allowing analytics to be performed directly on a single copy of the data. This approach simplifies data governance and security, reduces functional silos and makes collaboration easier. By instilling high trust in data, organizations can operate with confidence and better understand how data is acquired, changed, used and impacted across every analytics workload.A data governance solution for a data lakehouse provides a number of key capabilities:Centralized data catalog: A centralized data catalog stores all your data, ML models and analytics artifacts as well as metadata for each object. The unified catalog also blends in data from other catalogs such as an existing Hive metastore.Unified data access controls: A single and unified permissions model across all assets and all clouds. This includes attribute-based access control (ABAC) for personally identifiable information (PII).Data auditing: Data access is centrally audited with alerts and monitoring capabilities to promote accountability and securityData quality management: Robust data quality management with built-in quality controls, testing, monitoring and enforcement to ensure accurate and useful data is availableData lineage: Data lineage to get end-to-end visibility into how data flows in the lakehouse, from source to consumption, down to the column levelData discovery: Easy data discovery to enable data scientists, analysts, engineers and stakeholders to quickly discover and reference relevant data and accelerate time to valueData sharing and collaboration: Data can be shared — with fine-grained access controls — across clouds, regions and platforms, preventing silos from formingData cleanrooms for privacy-safe collaboration: Collaborate on sensitive data with internal or external stakeholders in a privacy preserving environmentOpen marketplace for data, analytics and AI: Discover, access and deploy data sets, as well as AI and analytical assets — such as ML models, notebooks, applications and dashboards — without proprietary platform dependencies, complicated ETL or expensive replication.Who oversees data governance?Chief Data OfficerYour Chief Data Officer (CDO) is the most senior executive on your governance team. Ultimately, they’re responsible for your data’s security, accessibility and usability.A CDO’s role involves setting the system up, securing funding and staff for its operation (and for related aspects like tools to automate some processes), and performing regular checks on its overall status.Data ownersData owners are individuals or teams responsible for the technical administration of your data sets. They might make decisions on which team members should have access to which kinds of information. If their policies (or lack thereof) lead to a data breach, they could be held accountable.To fulfill this role and its many responsibilities, data owners are typically also senior members of your organization.Data stewardsTo assist in the day-to-day running of your data governance workflows, data owners and CDOs will appoint data stewards. Data stewardship essentially involves implementing the program that has been set out for them, and ensuring both old and new data is managed appropriately. They’re responsible for monitoring compliance from both employees and customers, and escalating issues if they arise.Data governance committeesThis committee will be the main body that creates relevant policies in your organization.Often, it will consist of senior executives and data owners, who have a keen interest in the security and usability of data. Once their policies have been approved, they may set out procedures for stewards to follow, and also resolve disputes between parties.What is the difference between data management and data governance?Effective data management and data governance are crucial for any organization that handles data. Despite the frequent interchangeable use of these terms, they have distinct differences. Data management focuses on the technical aspects of data lifecycle management, including data ingestion, integration, organization, transformation and persistence, such as backup, retrieval and archiving. In contrast, data governance is about defining organizational policies, frameworks and tools to ensure that data-related requirements are aligned with the business strategy. This includes data accuracy, consistency, compliance with regulations, and internal organizational policies as well as data quality, security, privacy, auditing and risk management. Furthermore, data governance involves defining data ownership, roles and responsibilities, and enforcing policies and procedures throughout the organization. As a key pillar of a long-term data strategy that leverages data as a strategic asset, data governance plays a significant role, while data management deals with the operational aspect of delivering on that strategy.Learn more about data governance and data sharing on DatabricksUnity CatalogData SharingDatabricks MarketplaceCleanroomsFAQWhat are the main steps of data governance?The principal steps required include:Identifying all sources of dataPreparing metadata for the data and organizing a metadata storage optionSetting up mechanisms to track data lineage for data flow and usageScanning for sensitive data in your data estateCreating a governance framework, with set policies and procedures for the various overseersChecking the framework’s compliance with relevant regulatory instrumentsPerforming data quality checks by creating a rules library that is centrally managed and versioned. Update the rules library periodically with new rules.Regularly audit data entitlements and access to ensure complianceIdentifying further risks, such as lack of data security or excess access to sensitive data setsSecuring the go-ahead from the senior executive team and data governance committeeHiring or training essential staff members, such as data stewardsEstablishing a means of data distribution, so everyone in your organization can locate the centralized data catalogHaving regular reviews with senior executives (inviting team member feedback)Constantly adapting the governance model, such as when new data sets are introducedOften, the CDO would begin creating a framework to follow these steps, with additional actors assisting them as the project grows in size.What are some basic data governance principles?Several principles are associated with sound data governance. These ideas developed in the field of data ethics and, today, many are enshrined in regulatory laws. Here are three of the most important:Accountability: This is important from two perspectives. First, you need accountability when setting up and maintaining a data governance framework. Each agent, steward and owner must know what their task is and how they can perform it. Second, if something does go wrong, there must be a clear process for holding that individual/team to account and ensuring it does not happen again.Standardization: Understanding and following regulatory laws is key to creating a successful strategy. These rules on data security and storage should be followed by businesses as standard practice to ensure your data framework may operate universally across legal jurisdictions.Transparency: Data governance has everything to do with managing other people’s data, such as your employees, clients or customers. You must make it clear to everyone what your policies are, what data is collected and why.What is at the core of data governance?As an organization, the democratization of data, analytics and AI is at the core of data governance. It’s about making data accessible — and usable — throughout your organization, without compromising on quality or security. Your clients and customers should have full confidence that their information is kept in accordance with regulations, while your teams can make the best use of all the data you’ve gathered. This blend of accessibility and security is at the heart of data governance.ResourcesKeynoteData Governance and Sharing on the Lakehouse at Data + AI Summit 2022WebinarHow to Simplify Data and AI GovernanceeBooksData, Analytics and AI GovernanceA New Approach to Secure Data SharingProductPlatform OverviewPricingOpen Source TechTry DatabricksDemoProductPlatform OverviewPricingOpen Source TechTry DatabricksDemoLearn & SupportDocumentationGlossaryTraining & CertificationHelp CenterLegalOnline CommunityLearn & SupportDocumentationGlossaryTraining & CertificationHelp CenterLegalOnline CommunitySolutionsBy IndustriesProfessional ServicesSolutionsBy IndustriesProfessional ServicesCompanyAbout UsCareers at DatabricksDiversity and InclusionCompany BlogContact UsCompanyAbout UsCareers at DatabricksDiversity and InclusionCompany BlogContact UsSee Careers
at DatabricksWorldwideEnglish (United States)Deutsch (Germany)Français (France)Italiano (Italy)日本語 (Japan)한국어 (South Korea)Português (Brazil)Databricks Inc.
160 Spear Street, 13th Floor
San Francisco, CA 94105
1-866-330-0121© Databricks 2023. All rights reserved. Apache, Apache Spark, Spark and the Spark logo are trademarks of the Apache Software Foundation.Privacy Notice|Terms of Use|Your Privacy Choices|Your California Privacy Rights