Package: midi-etl
MIDI ETLThis repository contains an implementation of a modern data stack for building and analyzing MIDI datasets at scale. The stack is composed of several open source technologies, including DBT, Dagster, Trino, and Minio.DBT (Data Build Tool) is used to transform and optimize MIDI data as it is ingested from various sources into a data warehouse. DBT scripts are used to clean and preprocess the data, extract relevant metadata, and load it into a staging table.Dagster is used to define and execute data pipelines that fetch MIDI data from online sources using web scraping and API calls, and then load the data into the staging table in the data warehouse. It provides a framework for building, testing, and deploying these pipelines in a robust and scalable way.Trino is used to analyze the MIDI datasets stored in the data warehouse. It provides a distributed SQL query engine that can handle complex queries and large volumes of data efficiently. This allows data analysts and scientists to extract insights and trends from the MIDI datasets.Minio is used to store and retrieve the MIDI datasets and other data used in the stack. It is a lightweight, scalable object storage solution that can handle large volumes of data.Overall, this data stack provides a powerful and scalable platform for building and analyzing MIDI datasets. It can be used by data engineers, data scientists, and data analysts to extract insights and trends from music data at scale.PrerequisitesBefore you can use this repository, you will need to install the following:Docker : Follow the instructions athttps://docs.docker.com/get-docker/to install Docker on your machine.Docker Compose: follow the instructions athttps://docs.docker.com/compose/install/to install Docker Compose.InstallationYou can installmidi_etlusing pip:pip install midi_etlUsageTo use this repository, follow these steps:Clone the repository to your local machine:gitclonegit@gitlab.com:nintorac-audio/midi_etl.gitNavigate to the repository directory:cdmidi_etlBuild and start the Docker containers:docker-composeup--buildThis will build the Docker containers for the etl platform and deploy all the infra needed to run the projectNavigate todagitto initiate jobsDownload DBeaver (or your favourite DB IDE) to run queries over your data lake, navigate tominioto review the files in your data lake or use pyarrow to load the dataset in python eg.# First, import the necessary librariesimportpyarrowaspaimportpyarrow.parquetaspqimports3fsimportduckdb# Connect to Minio using s3fsfs=s3fs.S3FileSystem(anon=False,use_ssl=False,client_kwargs={"region_name":"us-east-1","endpoint_url":"http://localhost:9000","aws_access_key_id":"minio","aws_secret_access_key":"minio123","verify":False,})# Create a Parquet dataset for the path "midi_etl/midi" in the "datasets" bucketnote_ons=pq.ParquetDataset("midi_etl/midi/note_ons",filesystem=fs).read()# Open a connection to a DuckDB databaseconn=duckdb.connect()# Now you can run SQL queries on the table using the connectioncursor=conn.cursor()cursor.execute("SELECT * FROM note_ons LIMIT 10")print(cursor.fetchall())Makefileload_envis a make target that exports variables from a .env file into your local shellget_trino_cliis a make target that downloads the Trino command-line interface (CLI) from a URL. This is then mounted into the trino container to provide CLI Trino access.Available DatasetsLakh MIDI Dataset: The Lakh MIDI dataset is a collection of 176,581 MIDI files, 45,129 of which have been matched and aligned to entries in the Million Song Dataset, and is intended for use in large-scale music information retrievalLicenseThis repository is licensed under the MIT license. SeeLICENSEfor more information.VersioningFor now consider every version change a potentially breaking change. If I ever get around to adding tests
and such then I will add semantic versioning.