DATA LAKE Teradata Data Lake Solutions Keep original raw data fidelity WHAT IS A DATA LAKE? Raw data fidelity and long-term storage in the cloud. Data lakes and data warehouses are both design patterns, but they are opposites. Data warehouses structure and package data for the sake of quality, consistency, reuse, and performance with high concurrency. Data lakes complement warehouses with a design pattern that focuses on original raw data fidelity and long-term storage at a low cost while providing a new form of analytical agility. The Value in Data Lakes Data lake solutions meet the need to economically harness and derive value from exploding data volumes. This “dark” data from new sources—web, mobile, connected devices—was often discarded in the past, but it contains valuable insight. Massive volumes, plus new forms of analytics, demand a new way to manage and derive value from data. A data lake is a collection of long-term data containers that capture, refine, and explore any form of raw data at scale. It is enabled by low-cost technologies that multiple downstream facilities can draw upon, including data marts, data warehouses, and recommendation engines. Insights from Noncurated Data Prior to the big data trend, data integration normalized information in some sort of persistence – such as a database – and that created the value. This alone is no longer enough to manage all data in the enterprise, and attempting to structure it all undermines the value. That’s why dark data is rarely captured in a database, but data scientists often dig through dark data to find a few facts worth repeating. New Forms of Analytics The cloud era has given rise to new forms of analytics. Technologies like Apache Hadoop, Spark, and other innovations enable the parallelization of procedural programming languages, and this has enabled an entirely new breed of analytics. These new forms of analytics can be efficiently processed at scale, like graph, text, and machine learning algorithms that get an answer, then compare that answer to the next piece of data, and so on until a final output is reached. Corporate Memory Retention Archiving data that has not been used in a long time can save storage space in the data warehouse. Until the data lake design pattern came along, there was no other place to put colder data for occasional access except the high-performing data warehouse or the offline tape backup. With virtual query tools, users can easily access cold data in conjunction with the warm and hot data in the data warehouse through a single query. New Approach to Data Integration The industry has come full circle on how to best squeeze data transformation costs. The data lake solutions offer greater scalability than traditional ETL (extract, transform, load) servers at a lower cost. Organizations employing best practices are rebalancing hundreds of data integration jobs across the data lake, data warehouse, and ETL servers, as each has its own capabilities and economics. Common Pitfalls of Data Lakes On the surface, they appear straightforward—offering a way to manage and exploit massive volumes of structured and unstructured data. But, they are not as simple as they seem, and failed data lake projects are not uncommon across many types of industries and organizations. Early projects faced challenges because best practices had yet to emerge. Now a lack of solid design is the primary reason they don’t deliver their full value. Data silo and cluster proliferation. There is a notion that data lakes have a low barrier to entry and can be done makeshift in the cloud. This leads to redundant data and inconsistency with no two lakes reconciling, as well as synchronization problems. Lack of end user adoption. Users have the perception—right or wrong—that it’s too complicated to get answers from data lakes because it requires premium coding skills, or they can’t find the needles they need within the data haystacks. Limited commercial-off-the-shelf tools. Many vendors claim to connect to Hadoop or cloud object stores, but the offerings lack deep integration and most of these products were built for data warehouses, not data lakes. Conflicting objectives for data access. There is a balancing act between determining how strict security measures should be versus agile access. Plans and procedures need to be in place that align all stakeholders. The Data Lake Design Pattern The design pattern offers a set of workloads and expectations that guide a successful implementation. As technology and experience matured, an architecture and corresponding requirements evolved such that leading vendors have agreement and best practices for implementations. Technologies are critical, but the design pattern – which is independent of technology – is paramount. A data lake can be built on multiple technologies. While the Hadoop Distributed File System (HDFS) is what most people think of first, it is not required. TERADATA DATA LAKE SOLUTIONS Teradata Vantage, the platform for pervasive data intelligence, is designed to tap into the nuggets of information within customers’ data. The Teradata services team is well-versed in leveraging the many benefits of data lakes and related technologies such as Hadoop, Cassandra, and object stores like Amazon S3 and Azure Blob. AMAZON WEB SERVICES Use AWS infrastructure with Teradata Vantage MICROSOFT AZURE Combine Azure resources with Teradata Vantage GOOGLE CLOUD Leverage Google Cloud with Teradata Vantage Get started now with cloud analytics Rise above needless bottlenecks and complexity, take analytics to the cloud Download White Paper ALSO OF INTEREST The Data Lake is Dead; Long Live the Data Lake! What is a Data Lake? Data Warehouses vs Data Lakes PRODUCTS Vantage Overview Vantage Console 30-Day Free Trial Cloud Analytics Cloud Pricing Consulting Deployment Options View All Products INDUSTRIES Automotive Energy & Natural Resources Financial Services Government Healthcare & Life Sciences Manufacturing Retail Telco Travel & Transportation SOLUTIONS Asset Optimization Customer Experience Finance Transformation Operational Excellence Production Innovation Risk Mitigation See All Solutions TRENDS Artificial Intelligence Big Data Cloud Data Analytics Data Lake Data Management Data Science Data Warehouse IoT RESOURCES Analyst Content Blog Case Studies Certification Data Trends and Insights Glossary of Terms Library Webinars White Papers ABOUT About Teradata Careers Contact Us COVID-19 Response Customers Events Newsroom Partners Teradata Academics Global ©2021 Teradata. All Rights Reserved. Privacy Terms of Use Legal Support Policies Tracking Consent You Determine Your Experience As a data and analytics leader, Teradata will only collect, use, and track your personal data on our web properties when we have your permission. We want you to fully understand and have complete control over how your personal data is used by Teradata. Benefits of Sharing Your Data with Us Your personal data allows us to create a better online experience for you because it lets us tailor digital content to fit your needs and interests—Which products do you use? Which industry are you in? Learning these things helps us provide you with the most useful information. What Choices Do I Have? Your privacy is important to us, and your personal information is always handled in accordance with the Teradata Global Privacy Policy, which also explains our use of cookies and other technical means to optimize your site experience. By selecting one of the options below, you control your experience with Teradata web properties and acknowledge our tracking of your online behavior. Personalize My Online Experience You want a website that makes your life easier and content that is specific to your needs. Our goal is to deliver that. Your data allows our systems to remember what's important to you, so we can deliver content you want most, and save you time, effort, and frustration (we're nothing if not considerate). And, it can't be said enough: your privacy is important to us. Don't Personalize My Online Experience If you're content to have a less dynamic, non-personalized online experience with us, you can do it. Our Global Privacy Policy still applies, but because we don't use your information to personalize your experience on our web properties, you might spend more time looking for what you need. What If I Change My Mind? If you want to personalize your privacy settings in the future, click the "Tracking Consent" option in the footer. Submit