The free lunch is over, we have to ‘really’ deal with unstructured data
Channel: The ASF
Date: 13/12/2023

When Data Engineers think about unstructured data, basically the first idea that comes to mind is those pesky legacy files we need to transform to extract into some ‘good old’ structured table somewhere. But the recent improvements on Machine Learning and the growing popularity of Large Language Models (LLMs) have opened the pandora box of interest and requirements for Data Engineers. Users want to access and analyze data from unstructured data sources using natural language processing and we should maintain unstructured sources.

In this talk we will go into detail about what we need to do to get up to speed with the recent changes, we will talk about processing audio, images and text, using vector embeddings as well as the requirements for unstructured data based pipelines and how we can achieve them by relying on new open-source tools and our own good old Apache data engineering stack: Apache Spark, Lucene, Tika and others.

Speaker: Ismaël Mejía
Ismaël Mejía is a Senior Cloud Advocate at Microsoft working on the Azure Data team. He has more than a decade of experience architecting systems for startups and financial companies. He has been focused on distributed data and data engineering, he is a contributor to Apache Beam, Apache Avro and many other open-source projects. He is also a member of the Apache Software Foundation (ASF).