Package: getdaft
Website•Docs•Installation•10-minute tour of Daft•Community and SupportDaft: Distributed dataframes for multimodal dataDaftis a distributed query engine for large-scale data processing in Python and is implemented in Rust.Familiar interactive API:Lazy Python Dataframe for rapid and interactive iterationFocus on the what:Powerful Query Optimizer that rewrites queries to be as efficient as possibleData Catalog integrations:Full integration with data catalogs such as Apache IcebergRich multimodal type-system:Supports multimodal types such as Images, URLs, Tensors and moreSeamless Interchange: Built on theApache ArrowIn-Memory FormatBuilt for the cloud:Record-settingI/O performance for integrations with S3 cloud storageTable of ContentsAbout DaftGetting StartedBenchmarksRelated ProjectsLicenseAbout DaftDaft was designed with the following principles in mind:Any Data: Beyond the usual strings/numbers/dates, Daft columns can also hold complex or nested multimodal data such as Images, Embeddings and Python objects efficiently with it’s Arrow based memory representation. Ingestion and basic transformations of multimodal data is extremely easy and performant in Daft.Interactive Computing: Daft is built for the interactive developer experience through notebooks or REPLs - intelligent caching/query optimizations accelerates your experimentation and data exploration.Distributed Computing: Some workloads can quickly outgrow your local laptop’s computational resources - Daft integrates natively withRayfor running dataframes on large clusters of machines with thousands of CPUs/GPUs.Getting StartedInstallationInstall Daft withpip install getdaft.For more advanced installations (e.g. installing from source or with extra dependencies such as Ray and AWS utilities), please see ourInstallation GuideQuickstartCheck out our10-minute quickstart!In this example, we load images from an AWS S3 bucket’s URLs and resize each image in the dataframe:importdaft# Load a dataframe from filepaths in an S3 bucketdf=daft.from_glob_path("s3://daft-public-data/laion-sample-images/*")# 1. Download column of image URLs as a column of bytes# 2. Decode the column of bytes into a column of imagesdf=df.with_column("image",df["path"].url.download().image.decode())# Resize each image into 32x32df=df.with_column("resized",df["image"].image.resize(32,32))df.show(3)BenchmarksTo see the full benchmarks, detailed setup, and logs, check out ourbenchmarking page.More Resources10-minute tour of Daft- learn more about Daft’s full range of capabilities including dataloading from URLs, joins, user-defined functions (UDF), groupby, aggregations and more.User Guide- take a deep-dive into each topic within DaftAPI Reference- API reference for public classes/functions of DaftContributingTo start contributing to Daft, please readCONTRIBUTING.mdTelemetryTo help improve Daft, we collect non-identifiable data.To disable this behavior, set the following environment variable:DAFT_ANALYTICS_ENABLED=0The data that we collect is:Non-identifiable:events are keyed by a session ID which is generated on import of DaftMetadata-only:we do not collect any of our users’ proprietary code or dataFor development only:we do not buy or sell any user dataPlease see ourdocumentationfor more details.Related ProjectsDataframeQuery OptimizerMultimodalDistributedArrow BackedVectorized Execution EngineOut-of-coreDaftYesYesYesYesYesYesPandasNoPython objectNooptional >= 2.0Some(Numpy)NoPolarsYesPython objectNoYesYesYesModinEagarPython objectYesNoSome(Pandas)YesPysparkYesNoYesPandas UDF/IOPandas UDFYesDask DFNoPython objectYesNoSome(Pandas)YesCheck out ourdataframe comparison pagefor more details!LicenseDaft has an Apache 2.0 license - please see the LICENSE file.